Ir al contenido principal

Ralsina.Me — El sitio web de Roberto Alsina

Publicaciones sobre programming (publicaciones antiguas, página 95)

New Project: Hacé, like Make but lame

Since my dataflow li­brary Croupi­er is sort-of-­func­tion­al, I need­ed a project where I could ex­er­cise it.

This is im­por­tan­t, be­cause it's how you know if the de­sign of a li­brary is good, ex­ten­si­ble, and so on.

So, I de­cid­ed to write a min­i­mal make-­like-thing.

Well, good news, that was easy!

In about 50 lines of code I could write a thing that will run shell com­mands in a de­pen­den­cy dataflow!

It's called Hacé (don't both­er about how to pro­nounce it, I don't care) which is "im­per­a­tive make, sec­ond per­son sin­gu­lar" in ar­gen­tini­an span­ish, so it's an or­der to make.

I will spend a week or two mak­ing it in­to some­thing semi-use­ful, since it has some ad­van­tages over Make­files, such as re­act­ing to file con­tent and not file date, but its des­tiny is prob­a­bly just to be a test­bed for Croupi­er.

Croupier releases happened

A few re­leas­es of my Crys­tal task/­dataflow li­brary Croupi­er have gone out.

The main top­ic of work has been:

  • In­crease code qual­i­ty (I am still learn­ing the lan­guage af­ter al­l)
  • Make the API rea­son­able
  • Re­move re­stric­tions

On the lat­ter sub­jec­t, Croupi­er will now hap­pi­ly han­dle tasks with ze­ro or many in­put­s, with ze­ro or many out­put­s, or task that share some or all of their out­put­s, even if their in­puts dif­fer.

In all those cas­es it will try to Do The Right Thing, but it is ar­guable whether it does or not.

So, the API and what it can do is chang­ing of­ten. How­ev­er the ex­am­ple in the README on­ly need­ed one change from the first re­lease to now (be­cause it's pret­ty sim­ple)

I know no­body is ev­er go­ing to use it, it's a niche li­brary in a niche lan­guage, but I am hav­ing fun writ­ing it, and the con­cepts are quite wide­ly ap­pli­ca­ble, so it's ed­u­ca­tion­al.

New project: croupier

Intro to Dataflow Programming

This post is about ex­plain­ing a new pro­jec­t, called Croupi­er, which is a li­brary for dataflow pro­gram­ming.

What is that? It's a pro­gram­ming par­a­digm where you don't spec­i­fy the se­quence in which your code will ex­e­cute.

In­stead, you cre­ate a num­ber of "tasks", de­clare how the da­ta flows from one task to an­oth­er, pro­vide the ini­tial da­ta and then the sys­tem runs as many or as few of the tasks as need­ed, in what­ev­er or­der it deems bet­ter.

Examples

Put that way it looks scary and com­plex but it's some­thing so sim­ple al­most ev­ery pro­gram­mer has ran in­to a tool based on this prin­ci­ple:

make

When you create a Makefile, you declare a number of "targets", "dependencies" and "commands" (among other things) and then when you run make a_target it's make who decides which of those commands need to run, how and when.

Let's con­sid­er a more com­plex ex­am­ple: a stat­ic site gen­er­a­tor.

Usu­al­ly, these take a col­lec­tion of mark­down files with meta­da­ta such as ti­tle, date, tags, etc, and use that to pro­duce a col­lec­tion of HTML and oth­er files that con­sti­tute a web­site.

Now, let's con­sid­er it from the POV of dataflow pro­gram­ming with a sim­pli­fied ver­sion that on­ly takes mark­down files as in­puts and builds a "blog" out of them.

For each post in a file foo.md there will be a /foo.html.

But if that file has tags tag1 and tag2, then the contents of that file will affect the output files /tags/tag1.html and /tags/tag2.html

And if one of those tags is new, then it will affect tags/index.html

And if the post itself is new, then it will be in /index.html

And al­so in a RSS feed. And the RSS feeds for the tags!

As you can see, adding or mod­i­fy­ing a file can trig­ger a cas­cade of changes in the site.

Which you can mod­el as dataflow.

That's the ap­proach used by Niko­la, a stat­ic site gen­er­a­tor I wrote. Be­cause it's im­ple­ment­ed as dataflow, it can build on­ly what's need­ed, which in most cas­es is just a tiny frag­ment of the whole site.

That is done via doit an awe­some tool more peo­ple should know about, be­cause a lot more peo­ple should know about dataflow pro­gram­ming it­self.

So, what is Croupier?

It's a li­brary for dataflow pro­gram­ming in the Crys­tal lan­guage I am writ­ing!

Here's an ex­am­ple of it in use, from the doc­s, which should be self­-­ex­plana­to­ry if you have a pass­ing knowl­edge of Crys­tal or Ruby:

require "croupier"

b1 = ->{
  puts "task1 running"
  File.read("input.txt").downcase
}

Croupier::Task.new(
  name: "task1",
  output: "fileA",
  inputs: ["input.txt"],
  proc: b1
)

b2 = ->{
  puts "task2 running"
  File.read("fileA").upcase
}
Croupier::Task.new(
  name: "task2",
  output: "fileB",
  inputs: ["fileA"],
  proc: b2
)

Croupier::Task.run_tasks

Why?

Be­cause I want to write a fast SSG in Crys­tal, and be­cause dataflow pro­gram­ming is (to me) a fun­da­men­tal tool in my tool­kit.

Anything else?

I will prob­a­bly al­so do a sim­ple make-­like just as a play­ground for Croupi­er.

Nueva versión de mi sitio sobre nombres en Argentina

Des­pués de un lar­go tiem­po de te­ner­lo aban­do­na­do, le dí un po­co de ca­ri­ño a mi si­tio que ha­ce co­sas con in­for­ma­ción so­bre nom­bres de per­so­nas en Ar­gen­ti­na.

Un gráfico de barras mostrando nombres

En par­ti­cu­la­r:

  • Arre­glé una canti­dad de bugs
  • Hi­ce cam­bios en la ba­se de da­tos pa­ra que sea más rá­pi­do
  • Cam­bié los grá­fi­cos y aho­ra los ha­ce gnu­plot y se ven me­jor
  • Rees­cri­bí el có­di­go del ba­ckend en Cr­ys­tal por­que es­toy apren­dien­do ese len­gua­je.

Un gráfico mostrando como la popularidad de un nombre cambia

Pue­den ver la ver­sión nue­va en http­s://­nom­bres.­ral­si­na.­me y la ver­sión vie­ja en http­s://­ral­si­na.­me/s­to­rie­s/­nom­bres2/

Learning Crystal by Implementing a Static Site Generator

What?

A while back (10 YEARS???? WTH.) I wrote a stat­ic site gen­er­a­tor. I mean, I wrote one that is large and some­what pop­u­lar, called Niko­la but I al­so wrote a tiny one called Nico­let­ta

Why? Be­cause it's a nice lit­tle project and it shows the very ba­sics of how to do a whole projec­t.

All it does is:

  • Find mark­down files
  • Build them
  • Use tem­plates to gen­er­ate HTML files
  • Put those in an out­put fold­er

And that's it, that's a SS­G.

So, if I want­ed a "toy" project to prac­tice new (to me) pro­gram­ming lan­guages, why not re­write that?

And why not write about how it goes while I do it?

Hence this.

So, what's Crystal?

It's (they say) "A lan­guage for hu­mans and com­put­er­s". In short: a com­piled, stat­i­cal­ly typed lan­guage with a ru­by flavoured syn­tax.

And why? Again, why not?

Getting started

I in­stalled it us­ing curl and that got me ver­sion 1.8.2 which is the lat­est at the time of writ­ing this.

You can get your project start­ed by run­ning a com­mand:

nicoletta/crystal
✦ > crystal init app nicoletta .
    create  /home/ralsina/zig/nicoletta/crystal/.gitignore
    create  /home/ralsina/zig/nicoletta/crystal/.editorconfig
    create  /home/ralsina/zig/nicoletta/crystal/LICENSE
    create  /home/ralsina/zig/nicoletta/crystal/README.md
    create  /home/ralsina/zig/nicoletta/crystal/shard.yml
    create  /home/ralsina/zig/nicoletta/crystal/src/nicoletta.cr
    create  /home/ralsina/zig/nicoletta/crystal/spec/spec_helper.cr
    create  /home/ralsina/zig/nicoletta/crystal/spec/nicoletta_spec.cr
Initialized empty Git repository in /home/ralsina/zig/nicoletta/crystal/.git/

Some maybe in­ter­est­ing bit­s:

  • It inits a git re­po, with a git­ig­nore in it
  • Sets you up with a MIT li­cense
  • Cre­ates a rea­son­able README with nice place­hold­ers
  • We get a shard.ymlwith metadata
  • Source code in src/
  • spec/ seems to be for tests?

Mind you, I still have ze­ro idea about the lan­guage :-)

This ap­par­ent­ly com­piles in­to a do-noth­ing pro­gram, which is ok. Sur­prisied to see star­ship seems to sup­port crys­tal in the promp­t!

crystal on  main [?] is 📦 v0.1.0 via 🔮 v1.8.2 
> crystal build src/nicoletta.cr

crystal on  main [?] is 📦 v0.1.0 via 🔮 v1.8.2 
> ls -l
total 1748
-rw-rw-r-- 1 ralsina ralsina    2085 may 31 18:15 journal.md
-rw-r--r-- 1 ralsina ralsina    1098 may 31 18:08 LICENSE
-rwxrwxr-x 1 ralsina ralsina 1762896 may 31 18:15 nicoletta*
-rw-r--r-- 1 ralsina ralsina     604 may 31 18:08 README.md
-rw-r--r-- 1 ralsina ralsina     167 may 31 18:08 shard.yml
drwxrwxr-x 2 ralsina ralsina    4096 may 31 18:08 spec/
drwxrwxr-x 2 ralsina ralsina    4096 may 31 18:08 src/

Per­haps a bit sur­pris­ing that the do-noth­ing bi­na­ry is 1.7MB tho (1.2MB stripped) but it's just 380KB in "re­lease mod­e" which is nice.

Learning a Bit of Crystal

At this point I will stop and learn some syn­tax:

  • How to de­clare a vari­able / a lit­er­al / a con­stant
  • How to do an if / loop
  • How to de­fine / call a func­tion

Be­cause you know, one has to know at least that much 😁

There seems to be a de­cent set of tu­to­ri­als at this lev­el. let's see how it look­s.

Good thing: this is valid Crys­tal:

module Nicoletta
  VERSION = "0.1.0"

  😀 = "Hello world"
  puts 😀 
end

Al­so nice that vari­ables can change type.

Having the docs say integers are int32 and anything else is "for special use cases" is not great. int32 is small.

Al­so not a huge fan of sep­a­rate un­signed type­s.

I hate the "spaceship operator" <==> which "compares its operands and returns a value that is either zero (both operands are equal), a positive value (the first operand is bigger), or a negative value (the second operand is bigger)" ... hate it.

Num­bers have named meth­od­s, which is nice. How­ev­er it ran­dom­ly shows some weird syn­tax that has not been seen be­fore. One of these is not like the oth­er­s:

p! -5.abs,   # absolute value
  4.3.round, # round to nearest integer
  5.even?,   # odd/even check
  10.gcd(16) # greatest common divisor

Or maybe the ? is just part of the method name? Who knows! Not me!

Nice string in­ter­po­la­tion thingie.

name = "Crystal"
puts "Hello #{name}"

Why would anyone add an underscore method to strings? That's just weird.

Slices are reasonable, whatever[x..y] uses negative indexes for "from the right".

We have truthy val­ues, 0 is truthy, on­ly nil, false and null point­ers are fal­sy. Ok.

I strongly dislike using unless as a keyword instead of if with a negated condition. I consider that to be keyword proliferation and cutesy.

Meth­ods sup­port over­load­ing. Ok.

Ok, I know just enough Crys­tal to be slight­ly dan­ger­ous. Those feel like good tu­to­ri­al­s. Short, to the point, give you enough rope to ... make some­thing with rope, or what­ev­er.

Learning a Bit More Crystal

So: er­rors? Class­es? Block­s? How?

Class­es are pret­ty straight­for­ward ... ap­par­ent­ly they are a bit frowned up­on for per­for­mance rea­sons be­cause they are heap al­lo­cat­ed, but what­evs.

In­her­i­tance with method over­load­ing is not my cup of tea but 🤷

Exceptions are pretty simple but begin / rescue / else / ensure / end? Eek.

Also, I find that variables have nil type in the ensure block confusing.

Re­quir­ing files is not go­ing to be a prob­lem.

Blocks are in­ter­est­ing but I am not go­ing to try to use them yet.

Dinner Break

I will grab din­ner, and then try to im­ple­ment Nico­let­ta, some­how. I'll prob­a­bly fail 😅

Implementing Nicoletta

The code for nico­let­ta is not long so this should be a piece of cake.

No need to have a main in Crystal. Things just are executed.

First, I need a way to read the con­fig­u­ra­tion. It looks like this:

TITLE: "Nicoletta Test Blog"

That is tech­ni­cal­ly YAML so sure­ly there is a crys­tal thing to read it. In fac­t, it's in the stan­dard li­brary! This frag­ment work­s:

require "yaml"

VERSION = "0.1.0"

tpl_data = File.open("conf") do |file|
  YAML.parse(file)
end
p! tpl_data

And when ex­e­cut­ed does this, which is cor­rec­t:

crystal on  main [!?] is 📦 v0.1.0 via 🔮 v1.8.2 
> crystal run src/nicoletta.cr
tpl_data # => {"TITLE" => "Nicoletta Test Blog"}

Looks like what I want to store this sort of da­ta is a Hash

Next step: read tem­plates and put them in a hash in­dexed by path.

Templates are files in templates/ which look like this:

<h2><a href="${link}">${title}</a></h2>
date: ${date}
<hr>
${text}

Of course the syn­tax will prob­a­bly have to change, but for now I don't care.

To find all files in templates I can apparently use Dir.glob

And I swear I wrote this al­most in the first at­temp­t:

# Load templates
templates = {} of String => String
Dir.glob("templates/*.tmpl").each do |path|
  templates[path] = File.read(path)
end

Next is iterating over all files in posts/ (which are meant to be markdown with YAML metadata on top) and do things with them.

It­er­at­ing them is the same as be­fore (hey, this is nice)

Dir.glob("posts/*.md").each do |path|
  # Stuff
end

But I will need a Post class and so on, so...

Here is a Post class that is initialized by a path, parses metadata and keeps the text.

class Post
  def initialize(path)
    contents = File.read(path)
    metadata, @text = contents.split("\n\n", 2)
    @metadata = YAML.parse(metadata)
  end
  @metadata : YAML::Any
  @text : String
end

Next step is to give that class a method to parse the mark­down and con­vert it to HTM­L.

I am not im­ple­ment­ing that so I googled for a Crys­tal mark­down im­ple­men­ta­tion and found markd which is sad­ly aban­doned but looks ok.

Using it is surprisingly painless thanks to Crystal's shards dependency manager. First, I added it to shard.yml:

dependencies:
  markd:
   github: icyleaf/markd

Ran shards install:

crystal on  main [!+?] is 📦 v0.1.0 via 🔮 v1.8.2 
> shards install
Resolving dependencies
Fetching https://github.com/icyleaf/markd.git
Installing markd (0.5.0)
Writing shard.lock

Then added a require "markd", slapped this code in the Post class and that's it:

  def html
    Markd.to_html(@text)
  end

Here is the code to parse all the posts and hold them in an ar­ray:

posts = [] of Post

Dir.glob("posts/*.md").each do |path|
  posts << Post.new(path)
end

Now I need a Crys­tal im­ple­men­ta­tion of some tem­plate lan­guage, some­thing like han­dle­bars, I don't need much!

The stan­dard li­brary has a tem­plate lan­guage called ECR which is pret­ty nice but it's com­pile-­time and I need this to be done in run­time. So googled and found ... Kilt

I will use the crus­tache vari­ant, which im­ple­ments the Mus­tache stan­dard.

Again, added the dependency to shard.yml and ran shards install:

dependencies:
  markd:
   github: icyleaf/markd
  crustache:
   github: MakeNowJust/crustache

Af­ter some refac­tor­ing of tem­plate code, the tem­plate load­er now looks like this:

class Template
  @text : String
  @compiled : Crustache::Syntax::Template

  def initialize(path)
    @text = File.read(path)
    @compiled = Crustache.parse(@text)
  end
end

# Load templates
templates = {} of String => Template

Dir.glob("templates/*.tmpl").each do |path|
  templates[path] = Template.new(path)
end

I changed the tem­plates from what­ev­er they were be­fore to mus­tache:

<h2><a href="{{link}}">{{title}}</a></h2>
date: {{date}}
<hr>
{{text}}

I can now implement Post.render... except that top-level variables like templates are not accessible from inside classes and that messes up my code, so it needs refactoring. So.

This sure as hell is not id­iomat­ic Crys­tal, but bear with me, I am a be­gin­ner here.

This scans for all posts, then prints them rendered with the post.tmpl template:

class Post
  @metadata = {} of YAML::Any => YAML::Any
  @text : String
  @link : String
  @html : String

  def initialize(path)
    contents = File.read(path)
    metadata, @text = contents.split("\n\n", 2)
    @metadata = YAML.parse(metadata).as_h
    @link = path.split("/")[-1][0..-4] + ".html"
    @html = Markd.to_html(@text)
  end

  def render(template)
    Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html})
  end
end

posts = [] of Post

Dir.glob("posts/*.md").each do |path|
  posts << Post.new(path)
  p! p.render templates["templates/post.tmpl"]
end

Believe it or not, this is almost done. Now I need to make it output that (passed through another template) into the right path in a output/ folder.

This al­most work­s:

Dir.glob("posts/*.md").each do |path|
  post = Post.new(path)
  rendered_post = post.render templates["templates/post.tmpl"]
  rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled,
    tpl_data.merge({
      "content" => rendered_post,
    }))
  File.open("output/#{post.@link}", "w") do |io|
    io.puts rendered_page
  end
end

For some rea­son all my HTML is es­caped, I think that's the tem­plate en­gine try­ing to be safe 😤

Turns out I had to use TRIPLE han­dle­bars to print un­escaped HTM­L, so af­ter a small fix in the tem­plates...

A small HTML page

So, suc­cess! It has been fun, and I quite like the lan­guage!

I pub­lished it at my git serv­er but here's the full source code, all 60 lines of it:

# Nicoletta, a minimal static site generator.

require "yaml"
require "markd"
require "crustache"

VERSION = "0.1.0"

# Load config file
tpl_data = File.open("conf") do |file|
  YAML.parse(file).as_h
end

class Template
  @text : String
  @compiled : Crustache::Syntax::Template

  def initialize(path)
    @text = File.read(path)
    @compiled = Crustache.parse(@text)
  end
end

# Load templates
templates = {} of String => Template

Dir.glob("templates/*.tmpl").each do |path|
  templates[path] = Template.new(path)
end

class Post
  @metadata = {} of YAML::Any => YAML::Any
  @text : String
  @link : String
  @html : String

  def initialize(path)
    contents = File.read(path)
    metadata, @text = contents.split("\n\n", 2)
    @metadata = YAML.parse(metadata).as_h
    @link = path.split("/")[-1][0..-4] + ".html"
    @html = Markd.to_html(@text)
  end

  def render(template)
    Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html})
  end
end

Dir.glob("posts/*.md").each do |path|
  post = Post.new(path)
  rendered_post = post.render templates["templates/post.tmpl"]
  rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled,
    tpl_data.merge({
      "content" => rendered_post,
    }))
  File.open("output/#{post.@link}", "w") do |io|
    io.puts rendered_page
  end
end

Contents © 2000-2024 Roberto Alsina