Ir al contenido principal

Ralsina.Me — El sitio web de Roberto Alsina

Publicaciones sobre nikola

New Project: Shortcodes

One thing leads to an­oth­er. I want­ed to learn a new lan­guage and chose Crys­tal. To prac­tice, I rewrote my stan­dard prac­tice pro­jec­t, Nico­let­ta, a stat­ic site gen­er­a­tor.

That brought to mind doit a task/­dataflow li­brary, which made me want to im­ple­ment some­thing sim­i­lar, so I wrote Croupi­er which went well, so I said "Why not write a more se­ri­ous SS­G, which is Nicol­i­no which I think would not be com­plete with­out a fea­ture from Niko­la which I stole from Hugo, called short­codes. So I had to write a short­code pars­er.

What are short­codes?

Sort of macros for markdown. You want to do a <figure> but markdown doesn't have it, so you do this:

{{% figure src="http://whatever" %}}

And you end up with a fig­ure tag in your doc­u­men­t.

That is done by pass­ing da­ta like that "s­r­c" ar­gu­ment and the con­text of the doc­u­ment you are writ­ing in­to a tem­plate, which ren­ders a piece of HTM­L, that's in­sert­ed where the short­code was and bob's your un­cle.

There are vari­ants, and de­tail­s, and sub­tleties, but that's the ba­sic idea.

The bad news was that I was not the au­thor of Niko­la's cur­rent short­code parser, that was some­one else, and I re­al­ly don't un­der­stant their code.

So I want­ed to write it from scratch.

I have been want­ing to learn ragel for a bit, let's say 10 years. It's a thing that takes a fi­nite state ma­chine de­scrip­tion in a DSL and turns it in­to a pars­er in the lan­guage you wan­t.

As long as you want one of the lan­guages it likes, which is not Crys­tal, so I used it to gen­er­ate it in C. And wrapped that in Crys­tal.

So, that's a very long way to say that I wrote a Short­code pars­er li­brary that can be used from Crys­tal and from C, and since it can be used from C, it can be used from pret­ty much any lan­guage.

It does­n't im­ple­ment the whole short­code "spec­i­fi­ca­tion" (which is not a re­al thing that ex­ist­s) but it im­ple­ments the ba­sic­s.

Now, I just need to use it in Nicol­i­no, and even­tu­al­ly make it bet­ter as need­ed.

BTW: Ragel rocks.

Learning Crystal by Implementing a Static Site Generator

What?

A while back (10 YEARS???? WTH.) I wrote a stat­ic site gen­er­a­tor. I mean, I wrote one that is large and some­what pop­u­lar, called Niko­la but I al­so wrote a tiny one called Nico­let­ta

Why? Be­cause it's a nice lit­tle project and it shows the very ba­sics of how to do a whole projec­t.

All it does is:

  • Find mark­down files
  • Build them
  • Use tem­plates to gen­er­ate HTML files
  • Put those in an out­put fold­er

And that's it, that's a SS­G.

So, if I want­ed a "toy" project to prac­tice new (to me) pro­gram­ming lan­guages, why not re­write that?

And why not write about how it goes while I do it?

Hence this.

So, what's Crystal?

It's (they say) "A lan­guage for hu­mans and com­put­er­s". In short: a com­piled, stat­i­cal­ly typed lan­guage with a ru­by flavoured syn­tax.

And why? Again, why not?

Getting started

I in­stalled it us­ing curl and that got me ver­sion 1.8.2 which is the lat­est at the time of writ­ing this.

You can get your project start­ed by run­ning a com­mand:

nicoletta/crystal
✦ > crystal init app nicoletta .
    create  /home/ralsina/zig/nicoletta/crystal/.gitignore
    create  /home/ralsina/zig/nicoletta/crystal/.editorconfig
    create  /home/ralsina/zig/nicoletta/crystal/LICENSE
    create  /home/ralsina/zig/nicoletta/crystal/README.md
    create  /home/ralsina/zig/nicoletta/crystal/shard.yml
    create  /home/ralsina/zig/nicoletta/crystal/src/nicoletta.cr
    create  /home/ralsina/zig/nicoletta/crystal/spec/spec_helper.cr
    create  /home/ralsina/zig/nicoletta/crystal/spec/nicoletta_spec.cr
Initialized empty Git repository in /home/ralsina/zig/nicoletta/crystal/.git/

Some maybe in­ter­est­ing bit­s:

  • It inits a git re­po, with a git­ig­nore in it
  • Sets you up with a MIT li­cense
  • Cre­ates a rea­son­able README with nice place­hold­ers
  • We get a shard.ymlwith metadata
  • Source code in src/
  • spec/ seems to be for tests?

Mind you, I still have ze­ro idea about the lan­guage :-)

This ap­par­ent­ly com­piles in­to a do-noth­ing pro­gram, which is ok. Sur­prisied to see star­ship seems to sup­port crys­tal in the promp­t!

crystal on  main [?] is 📦 v0.1.0 via 🔮 v1.8.2 
> crystal build src/nicoletta.cr

crystal on  main [?] is 📦 v0.1.0 via 🔮 v1.8.2 
> ls -l
total 1748
-rw-rw-r-- 1 ralsina ralsina    2085 may 31 18:15 journal.md
-rw-r--r-- 1 ralsina ralsina    1098 may 31 18:08 LICENSE
-rwxrwxr-x 1 ralsina ralsina 1762896 may 31 18:15 nicoletta*
-rw-r--r-- 1 ralsina ralsina     604 may 31 18:08 README.md
-rw-r--r-- 1 ralsina ralsina     167 may 31 18:08 shard.yml
drwxrwxr-x 2 ralsina ralsina    4096 may 31 18:08 spec/
drwxrwxr-x 2 ralsina ralsina    4096 may 31 18:08 src/

Per­haps a bit sur­pris­ing that the do-noth­ing bi­na­ry is 1.7MB tho (1.2MB stripped) but it's just 380KB in "re­lease mod­e" which is nice.

Learning a Bit of Crystal

At this point I will stop and learn some syn­tax:

  • How to de­clare a vari­able / a lit­er­al / a con­stant
  • How to do an if / loop
  • How to de­fine / call a func­tion

Be­cause you know, one has to know at least that much 😁

There seems to be a de­cent set of tu­to­ri­als at this lev­el. let's see how it look­s.

Good thing: this is valid Crys­tal:

module Nicoletta
  VERSION = "0.1.0"

  😀 = "Hello world"
  puts 😀 
end

Al­so nice that vari­ables can change type.

Having the docs say integers are int32 and anything else is "for special use cases" is not great. int32 is small.

Al­so not a huge fan of sep­a­rate un­signed type­s.

I hate the "spaceship operator" <==> which "compares its operands and returns a value that is either zero (both operands are equal), a positive value (the first operand is bigger), or a negative value (the second operand is bigger)" ... hate it.

Num­bers have named meth­od­s, which is nice. How­ev­er it ran­dom­ly shows some weird syn­tax that has not been seen be­fore. One of these is not like the oth­er­s:

p! -5.abs,   # absolute value
  4.3.round, # round to nearest integer
  5.even?,   # odd/even check
  10.gcd(16) # greatest common divisor

Or maybe the ? is just part of the method name? Who knows! Not me!

Nice string in­ter­po­la­tion thingie.

name = "Crystal"
puts "Hello #{name}"

Why would anyone add an underscore method to strings? That's just weird.

Slices are reasonable, whatever[x..y] uses negative indexes for "from the right".

We have truthy val­ues, 0 is truthy, on­ly nil, false and null point­ers are fal­sy. Ok.

I strongly dislike using unless as a keyword instead of if with a negated condition. I consider that to be keyword proliferation and cutesy.

Meth­ods sup­port over­load­ing. Ok.

Ok, I know just enough Crys­tal to be slight­ly dan­ger­ous. Those feel like good tu­to­ri­al­s. Short, to the point, give you enough rope to ... make some­thing with rope, or what­ev­er.

Learning a Bit More Crystal

So: er­rors? Class­es? Block­s? How?

Class­es are pret­ty straight­for­ward ... ap­par­ent­ly they are a bit frowned up­on for per­for­mance rea­sons be­cause they are heap al­lo­cat­ed, but what­evs.

In­her­i­tance with method over­load­ing is not my cup of tea but 🤷

Exceptions are pretty simple but begin / rescue / else / ensure / end? Eek.

Also, I find that variables have nil type in the ensure block confusing.

Re­quir­ing files is not go­ing to be a prob­lem.

Blocks are in­ter­est­ing but I am not go­ing to try to use them yet.

Dinner Break

I will grab din­ner, and then try to im­ple­ment Nico­let­ta, some­how. I'll prob­a­bly fail 😅

Implementing Nicoletta

The code for nico­let­ta is not long so this should be a piece of cake.

No need to have a main in Crystal. Things just are executed.

First, I need a way to read the con­fig­u­ra­tion. It looks like this:

TITLE: "Nicoletta Test Blog"

That is tech­ni­cal­ly YAML so sure­ly there is a crys­tal thing to read it. In fac­t, it's in the stan­dard li­brary! This frag­ment work­s:

require "yaml"

VERSION = "0.1.0"

tpl_data = File.open("conf") do |file|
  YAML.parse(file)
end
p! tpl_data

And when ex­e­cut­ed does this, which is cor­rec­t:

crystal on  main [!?] is 📦 v0.1.0 via 🔮 v1.8.2 
> crystal run src/nicoletta.cr
tpl_data # => {"TITLE" => "Nicoletta Test Blog"}

Looks like what I want to store this sort of da­ta is a Hash

Next step: read tem­plates and put them in a hash in­dexed by path.

Templates are files in templates/ which look like this:

<h2><a href="${link}">${title}</a></h2>
date: ${date}
<hr>
${text}

Of course the syn­tax will prob­a­bly have to change, but for now I don't care.

To find all files in templates I can apparently use Dir.glob

And I swear I wrote this al­most in the first at­temp­t:

# Load templates
templates = {} of String => String
Dir.glob("templates/*.tmpl").each do |path|
  templates[path] = File.read(path)
end

Next is iterating over all files in posts/ (which are meant to be markdown with YAML metadata on top) and do things with them.

It­er­at­ing them is the same as be­fore (hey, this is nice)

Dir.glob("posts/*.md").each do |path|
  # Stuff
end

But I will need a Post class and so on, so...

Here is a Post class that is initialized by a path, parses metadata and keeps the text.

class Post
  def initialize(path)
    contents = File.read(path)
    metadata, @text = contents.split("\n\n", 2)
    @metadata = YAML.parse(metadata)
  end
  @metadata : YAML::Any
  @text : String
end

Next step is to give that class a method to parse the mark­down and con­vert it to HTM­L.

I am not im­ple­ment­ing that so I googled for a Crys­tal mark­down im­ple­men­ta­tion and found markd which is sad­ly aban­doned but looks ok.

Using it is surprisingly painless thanks to Crystal's shards dependency manager. First, I added it to shard.yml:

dependencies:
  markd:
   github: icyleaf/markd

Ran shards install:

crystal on  main [!+?] is 📦 v0.1.0 via 🔮 v1.8.2 
> shards install
Resolving dependencies
Fetching https://github.com/icyleaf/markd.git
Installing markd (0.5.0)
Writing shard.lock

Then added a require "markd", slapped this code in the Post class and that's it:

  def html
    Markd.to_html(@text)
  end

Here is the code to parse all the posts and hold them in an ar­ray:

posts = [] of Post

Dir.glob("posts/*.md").each do |path|
  posts << Post.new(path)
end

Now I need a Crys­tal im­ple­men­ta­tion of some tem­plate lan­guage, some­thing like han­dle­bars, I don't need much!

The stan­dard li­brary has a tem­plate lan­guage called ECR which is pret­ty nice but it's com­pile-­time and I need this to be done in run­time. So googled and found ... Kilt

I will use the crus­tache vari­ant, which im­ple­ments the Mus­tache stan­dard.

Again, added the dependency to shard.yml and ran shards install:

dependencies:
  markd:
   github: icyleaf/markd
  crustache:
   github: MakeNowJust/crustache

Af­ter some refac­tor­ing of tem­plate code, the tem­plate load­er now looks like this:

class Template
  @text : String
  @compiled : Crustache::Syntax::Template

  def initialize(path)
    @text = File.read(path)
    @compiled = Crustache.parse(@text)
  end
end

# Load templates
templates = {} of String => Template

Dir.glob("templates/*.tmpl").each do |path|
  templates[path] = Template.new(path)
end

I changed the tem­plates from what­ev­er they were be­fore to mus­tache:

<h2><a href="{{link}}">{{title}}</a></h2>
date: {{date}}
<hr>
{{text}}

I can now implement Post.render... except that top-level variables like templates are not accessible from inside classes and that messes up my code, so it needs refactoring. So.

This sure as hell is not id­iomat­ic Crys­tal, but bear with me, I am a be­gin­ner here.

This scans for all posts, then prints them rendered with the post.tmpl template:

class Post
  @metadata = {} of YAML::Any => YAML::Any
  @text : String
  @link : String
  @html : String

  def initialize(path)
    contents = File.read(path)
    metadata, @text = contents.split("\n\n", 2)
    @metadata = YAML.parse(metadata).as_h
    @link = path.split("/")[-1][0..-4] + ".html"
    @html = Markd.to_html(@text)
  end

  def render(template)
    Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html})
  end
end

posts = [] of Post

Dir.glob("posts/*.md").each do |path|
  posts << Post.new(path)
  p! p.render templates["templates/post.tmpl"]
end

Believe it or not, this is almost done. Now I need to make it output that (passed through another template) into the right path in a output/ folder.

This al­most work­s:

Dir.glob("posts/*.md").each do |path|
  post = Post.new(path)
  rendered_post = post.render templates["templates/post.tmpl"]
  rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled,
    tpl_data.merge({
      "content" => rendered_post,
    }))
  File.open("output/#{post.@link}", "w") do |io|
    io.puts rendered_page
  end
end

For some rea­son all my HTML is es­caped, I think that's the tem­plate en­gine try­ing to be safe 😤

Turns out I had to use TRIPLE han­dle­bars to print un­escaped HTM­L, so af­ter a small fix in the tem­plates...

A small HTML page

So, suc­cess! It has been fun, and I quite like the lan­guage!

I pub­lished it at my git serv­er but here's the full source code, all 60 lines of it:

# Nicoletta, a minimal static site generator.

require "yaml"
require "markd"
require "crustache"

VERSION = "0.1.0"

# Load config file
tpl_data = File.open("conf") do |file|
  YAML.parse(file).as_h
end

class Template
  @text : String
  @compiled : Crustache::Syntax::Template

  def initialize(path)
    @text = File.read(path)
    @compiled = Crustache.parse(@text)
  end
end

# Load templates
templates = {} of String => Template

Dir.glob("templates/*.tmpl").each do |path|
  templates[path] = Template.new(path)
end

class Post
  @metadata = {} of YAML::Any => YAML::Any
  @text : String
  @link : String
  @html : String

  def initialize(path)
    contents = File.read(path)
    metadata, @text = contents.split("\n\n", 2)
    @metadata = YAML.parse(metadata).as_h
    @link = path.split("/")[-1][0..-4] + ".html"
    @html = Markd.to_html(@text)
  end

  def render(template)
    Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html})
  end
end

Dir.glob("posts/*.md").each do |path|
  post = Post.new(path)
  rendered_post = post.render templates["templates/post.tmpl"]
  rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled,
    tpl_data.merge({
      "content" => rendered_post,
    }))
  File.open("output/#{post.@link}", "w") do |io|
    io.puts rendered_page
  end
end

Using scripts to update my personal site.

These last few days I have been adding code in Niko­la to give it a more use­ful Python API. I added scripts then I start­ed a PR that lets you mod­i­fy posts pro­gram­mat­i­cal­ly.

Why?

Be­cause my site has 20 years of bag­gage. Which means ev­ery bad idea in the my 20 year his­to­ry of do­ing my own blog soft­ware is lurk­ing in it some­where.

For ex­am­ple, when Niko­la got start­ed, it had (it still has it!) sup­port for what I called "meta files". Ba­si­cal­ly, you put your post's con­tent in a file, say "my­post.tx­t" and you added things like the date, the ti­tle, tags and so on in "my­post.meta", which was the metafile.

That was good in that it was a way to quick­ly get it work­ing with­out wor­ry­ing about how to ex­tract meta­da­ta from source files, and to keep source files com­pat­i­ble with oth­er toolchain­s, like do­cu­til­s' or nor­mal mark­down.

BUT, then we added ways to have metadata in the files and keep them compatible. But I still had 1500 metafiles in my site. And getting rid of them would involve some sed some python and some pain, so I never upgraded the posts to the newer format.

Un­til now.

two_post_files = [p for p in site.timeline if p.is_two_file]

for p in two_post_files:
    p.is_two_file = False
    print(p.title())

What is that? Well, it filters the site.timeline and finds all the things that are in two files using the is_two_file property, and then makes them not be two files.

What is the re­sult?

$ git diff --stat 766d8e1c5dd495d4aa7e27bb0b7f6b2c62c6aa63 | tail -1
 3739 files changed, 20521 insertions(+), 7381 deletions(-)

Of course my site is under git, I would not dare do this without it. And hey, no more .meta files!

Server Architecture

Just for the record, I saw this hi­lar­i­ous tweet about what you need to "prop­er­ly" de­ploy Word­press in AWS:

So I quick­ly got my Leno­vo Pen out of its hold­er and pro­ceed­ed to do the equiv­a­lent for Niko­la:

/images/arch.png

Yes, that is my hand­writ­ing.

New Nikola Thing: scripts

Niko­la is a stat­ic site gen­er­a­tor, and it knows its au­di­ence: Nerd­s, pro­gram­mer­s, sci­ence peo­ple, and the like. Oh, and me. I most­ly de­vel­op it for me.

One im­por­tant thing for this cat­e­go­ry of tools is that they should cater to what the users want to do, and al­so to how they want to do it.

So, faced with the need to do things like "set this spe­cif­ic meta­da­ta field in these 490 posts out of the 1450 you have" ... edit­ing them man­u­al­ly is not go­ing to hap­pen.

Sure, I could sed/python/what­ev­er my way to do it "au­to­mat­i­cal­ly". But that is go­ing to be aw­ful­ly er­ror prone.

So, I have start­ed a cam­paign to fix it. I want to make Niko­la be the API to its da­ta. This has two sides.

I need to be able to run one-off things

This needs to be easier than creating a Nikola command plugin but less annoying than typing them in nikola console

[ralsina@salma static]$ nikola console
Scanning posts..........done!
Python 3.8.2 (default, Feb 26 2020, 22:21:03) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.


Nikola v8.0.4 -- IPython Console (conf = configuration file, site, nikola_site = site engine, commands = nikola commands)

In [1]:  

The good news is: this is done in git mas­ter!

Now you can create a python script and run it as nikola console -s cool_script.py and the script runs in the same context as the console, so you magically have the site itself and the timeline and the configuration and all the good stuff ready to use.

I need the API to be useful

And this is where Niko­la has ... not been a good boy. Since it was meant to gen­er­ate stat­ic sites, it's pret­ty good about of­fer­ing you ways to know things about your da­ta.

Want to know what is the de­scrip­tion of the tags ap­plied to the post in slovene? it to­tal­ly can do that in two lines.

Want to add a tag to a post? Sor­ry dude, that's im­pos­si­ble.

So, I am adding these things, slow­ly.

And that's the cur­rent sta­tus.


Contents © 2000-2023 Roberto Alsina