Full Changelog: v0.1.6...v0.1.7
Since my dataflow library Croupier is sort-of-functional, I needed a project where I could exercise it.
This is important, because it's how you know if the design of a library is good, extensible, and so on.
So, I decided to write a minimal make-like-thing.
Well, good news, that was easy!
In about 50 lines of code I could write a thing that will run shell commands in a dependency dataflow!
It's called Hacé (don't bother about how to pronounce it, I don't care) which is "imperative make, second person singular" in argentinian spanish, so it's an order to make.
I will spend a week or two making it into something semi-useful, since it has some advantages over Makefiles, such as reacting to file content and not file date, but its destiny is probably just to be a testbed for Croupier.
A few releases of my Crystal task/dataflow library Croupier have gone out.
The main topic of work has been:
On the latter subject, Croupier will now happily handle tasks with zero or many inputs, with zero or many outputs, or task that share some or all of their outputs, even if their inputs differ.
In all those cases it will try to Do The Right Thing, but it is arguable whether it does or not.
So, the API and what it can do is changing often. However the example in the README only needed one change from the first release to now (because it's pretty simple)
I know nobody is ever going to use it, it's a niche library in a niche language, but I am having fun writing it, and the concepts are quite widely applicable, so it's educational.
This post is about explaining a new project, called Croupier, which is a library for dataflow programming.
What is that? It's a programming paradigm where you don't specify the sequence in which your code will execute.
Instead, you create a number of "tasks", declare how the data flows from one task to another, provide the initial data and then the system runs as many or as few of the tasks as needed, in whatever order it deems better.
Put that way it looks scary and complex but it's something so simple almost every programmer has ran into a tool based on this principle:
make
When you create a Makefile, you declare a number of "targets", "dependencies" and "commands" (among other things) and then when you run make a_target it's make who decides which of those commands need to run, how and when.
Let's consider a more complex example: a static site generator.
Usually, these take a collection of markdown files with metadata such as title, date, tags, etc, and use that to produce a collection of HTML and other files that constitute a website.
Now, let's consider it from the POV of dataflow programming with a simplified version that only takes markdown files as inputs and builds a "blog" out of them.
For each post in a file foo.md there will be a /foo.html.
But if that file has tags tag1 and tag2, then the contents of that file will affect the output files /tags/tag1.html and /tags/tag2.html
And if one of those tags is new, then it will affect tags/index.html
And if the post itself is new, then it will be in /index.html
And also in a RSS feed. And the RSS feeds for the tags!
As you can see, adding or modifying a file can trigger a cascade of changes in the site.
Which you can model as dataflow.
That's the approach used by Nikola, a static site generator I wrote. Because it's implemented as dataflow, it can build only what's needed, which in most cases is just a tiny fragment of the whole site.
That is done via doit an awesome tool more people should know about, because a lot more people should know about dataflow programming itself.
It's a library for dataflow programming in the Crystal language I am writing!
Here's an example of it in use, from the docs, which should be self-explanatory if you have a passing knowledge of Crystal or Ruby:
require "croupier"
b1 = ->{
puts "task1 running"
File.read("input.txt").downcase
}
Croupier::Task.new(
name: "task1",
output: "fileA",
inputs: ["input.txt"],
proc: b1
)
b2 = ->{
puts "task2 running"
File.read("fileA").upcase
}
Croupier::Task.new(
name: "task2",
output: "fileB",
inputs: ["fileA"],
proc: b2
)
Croupier::Task.run_tasks
Because I want to write a fast SSG in Crystal, and because dataflow programming is (to me) a fundamental tool in my toolkit.
I will probably also do a simple make-like just as a playground for Croupier.
A while back (10 YEARS???? WTH.) I wrote a static site generator. I mean, I wrote one that is large and somewhat popular, called Nikola but I also wrote a tiny one called Nicoletta
Why? Because it's a nice little project and it shows the very basics of how to do a whole project.
All it does is:
And that's it, that's a SSG.
So, if I wanted a "toy" project to practice new (to me) programming languages, why not rewrite that?
And why not write about how it goes while I do it?
Hence this.
It's (they say) "A language for humans and computers". In short: a compiled, statically typed language with a ruby flavoured syntax.
And why? Again, why not?
I installed it using curl and that got me version 1.8.2 which is the latest at the time of writing this.
You can get your project started by running a command:
nicoletta/crystal
✦ > crystal init app nicoletta .
create /home/ralsina/zig/nicoletta/crystal/.gitignore
create /home/ralsina/zig/nicoletta/crystal/.editorconfig
create /home/ralsina/zig/nicoletta/crystal/LICENSE
create /home/ralsina/zig/nicoletta/crystal/README.md
create /home/ralsina/zig/nicoletta/crystal/shard.yml
create /home/ralsina/zig/nicoletta/crystal/src/nicoletta.cr
create /home/ralsina/zig/nicoletta/crystal/spec/spec_helper.cr
create /home/ralsina/zig/nicoletta/crystal/spec/nicoletta_spec.cr
Initialized empty Git repository in /home/ralsina/zig/nicoletta/crystal/.git/
Some maybe interesting bits:
shard.ymlwith metadatasrc/
spec/ seems to be for tests?Mind you, I still have zero idea about the language :-)
This apparently compiles into a do-nothing program, which is ok. Surprisied to see starship seems to support crystal in the prompt!
crystal on main [?] is 📦 v0.1.0 via 🔮 v1.8.2
> crystal build src/nicoletta.cr
crystal on main [?] is 📦 v0.1.0 via 🔮 v1.8.2
> ls -l
total 1748
-rw-rw-r-- 1 ralsina ralsina 2085 may 31 18:15 journal.md
-rw-r--r-- 1 ralsina ralsina 1098 may 31 18:08 LICENSE
-rwxrwxr-x 1 ralsina ralsina 1762896 may 31 18:15 nicoletta*
-rw-r--r-- 1 ralsina ralsina 604 may 31 18:08 README.md
-rw-r--r-- 1 ralsina ralsina 167 may 31 18:08 shard.yml
drwxrwxr-x 2 ralsina ralsina 4096 may 31 18:08 spec/
drwxrwxr-x 2 ralsina ralsina 4096 may 31 18:08 src/
Perhaps a bit surprising that the do-nothing binary is 1.7MB tho (1.2MB stripped) but it's just 380KB in "release mode" which is nice.
At this point I will stop and learn some syntax:
Because you know, one has to know at least that much 😁
There seems to be a decent set of tutorials at this level. let's see how it looks.
Good thing: this is valid Crystal:
module Nicoletta
VERSION = "0.1.0"
😀 = "Hello world"
puts 😀
end
Also nice that variables can change type.
Having the docs say integers are int32 and anything else is "for special use cases" is not great. int32 is small.
Also not a huge fan of separate unsigned types.
I hate the "spaceship operator" <==> which "compares its operands and returns a value that is either zero (both operands are equal), a positive value (the first operand is bigger), or a negative value (the second operand is bigger)" ... hate it.
Numbers have named methods, which is nice. However it randomly shows some weird syntax that has not been seen before. One of these is not like the others:
p! -5.abs, # absolute value
4.3.round, # round to nearest integer
5.even?, # odd/even check
10.gcd(16) # greatest common divisor
Or maybe the ? is just part of the method name? Who knows! Not me!
Nice string interpolation thingie.
name = "Crystal"
puts "Hello #{name}"
Why would anyone add an underscore method to strings? That's just weird.
Slices are reasonable, whatever[x..y] uses negative indexes for "from the right".
We have truthy values, 0 is truthy, only nil, false and null pointers are falsy. Ok.
I strongly dislike using unless as a keyword instead of if with a negated condition. I consider that to be keyword proliferation and cutesy.
Methods support overloading. Ok.
Ok, I know just enough Crystal to be slightly dangerous. Those feel like good tutorials. Short, to the point, give you enough rope to ... make something with rope, or whatever.
So: errors? Classes? Blocks? How?
Classes are pretty straightforward ... apparently they are a bit frowned upon for performance reasons because they are heap allocated, but whatevs.
Inheritance with method overloading is not my cup of tea but 🤷
Exceptions are pretty simple but begin / rescue / else / ensure / end? Eek.
Also, I find that variables have nil type in the ensure block confusing.
Requiring files is not going to be a problem.
Blocks are interesting but I am not going to try to use them yet.
I will grab dinner, and then try to implement Nicoletta, somehow. I'll probably fail 😅
The code for nicoletta is not long so this should be a piece of cake.
No need to have a main in Crystal. Things just are executed.
First, I need a way to read the configuration. It looks like this:
TITLE: "Nicoletta Test Blog"
That is technically YAML so surely there is a crystal thing to read it. In fact, it's in the standard library! This fragment works:
require "yaml"
VERSION = "0.1.0"
tpl_data = File.open("conf") do |file|
YAML.parse(file)
end
p! tpl_data
And when executed does this, which is correct:
crystal on main [!?] is 📦 v0.1.0 via 🔮 v1.8.2
> crystal run src/nicoletta.cr
tpl_data # => {"TITLE" => "Nicoletta Test Blog"}
Looks like what I want to store this sort of data is a Hash
Next step: read templates and put them in a hash indexed by path.
Templates are files in templates/ which look like this:
<h2><a href="${link}">${title}</a></h2>
date: ${date}
<hr>
${text}
Of course the syntax will probably have to change, but for now I don't care.
To find all files in templates I can apparently use Dir.glob
And I swear I wrote this almost in the first attempt:
# Load templates
templates = {} of String => String
Dir.glob("templates/*.tmpl").each do |path|
templates[path] = File.read(path)
end
Next is iterating over all files in posts/ (which are meant to be markdown with YAML metadata on top) and do things with them.
Iterating them is the same as before (hey, this is nice)
Dir.glob("posts/*.md").each do |path|
# Stuff
end
But I will need a Post class and so on, so...
Here is a Post class that is initialized by a path, parses metadata and keeps the text.
class Post
def initialize(path)
contents = File.read(path)
metadata, @text = contents.split("\n\n", 2)
@metadata = YAML.parse(metadata)
end
@metadata : YAML::Any
@text : String
end
Next step is to give that class a method to parse the markdown and convert it to HTML.
I am not implementing that so I googled for a Crystal markdown implementation and found markd which is sadly abandoned but looks ok.
Using it is surprisingly painless thanks to Crystal's shards dependency manager. First, I added it to shard.yml:
dependencies:
markd:
github: icyleaf/markd
Ran shards install:
crystal on main [!+?] is 📦 v0.1.0 via 🔮 v1.8.2
> shards install
Resolving dependencies
Fetching https://github.com/icyleaf/markd.git
Installing markd (0.5.0)
Writing shard.lock
Then added a require "markd", slapped this code in the Post class and that's it:
def html
Markd.to_html(@text)
end
Here is the code to parse all the posts and hold them in an array:
posts = [] of Post
Dir.glob("posts/*.md").each do |path|
posts << Post.new(path)
end
Now I need a Crystal implementation of some template language, something like handlebars, I don't need much!
The standard library has a template language called ECR which is pretty nice but it's compile-time and I need this to be done in runtime. So googled and found ... Kilt
I will use the crustache variant, which implements the Mustache standard.
Again, added the dependency to shard.yml and ran shards install:
dependencies:
markd:
github: icyleaf/markd
crustache:
github: MakeNowJust/crustache
After some refactoring of template code, the template loader now looks like this:
class Template
@text : String
@compiled : Crustache::Syntax::Template
def initialize(path)
@text = File.read(path)
@compiled = Crustache.parse(@text)
end
end
# Load templates
templates = {} of String => Template
Dir.glob("templates/*.tmpl").each do |path|
templates[path] = Template.new(path)
end
I changed the templates from whatever they were before to mustache:
<h2><a href="{{link}}">{{title}}</a></h2>
date: {{date}}
<hr>
{{text}}
I can now implement Post.render... except that top-level variables like templates are not accessible from inside classes and that messes up my code, so it needs refactoring. So.
This sure as hell is not idiomatic Crystal, but bear with me, I am a beginner here.
This scans for all posts, then prints them rendered with the post.tmpl template:
class Post
@metadata = {} of YAML::Any => YAML::Any
@text : String
@link : String
@html : String
def initialize(path)
contents = File.read(path)
metadata, @text = contents.split("\n\n", 2)
@metadata = YAML.parse(metadata).as_h
@link = path.split("/")[-1][0..-4] + ".html"
@html = Markd.to_html(@text)
end
def render(template)
Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html})
end
end
posts = [] of Post
Dir.glob("posts/*.md").each do |path|
posts << Post.new(path)
p! p.render templates["templates/post.tmpl"]
end
Believe it or not, this is almost done.
Now I need to make it output that (passed through another template) into the right path in a output/ folder.
This almost works:
Dir.glob("posts/*.md").each do |path|
post = Post.new(path)
rendered_post = post.render templates["templates/post.tmpl"]
rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled,
tpl_data.merge({
"content" => rendered_post,
}))
File.open("output/#{post.@link}", "w") do |io|
io.puts rendered_page
end
end
For some reason all my HTML is escaped, I think that's the template engine trying to be safe 😤
Turns out I had to use TRIPLE handlebars to print unescaped HTML, so after a small fix in the templates...
So, success! It has been fun, and I quite like the language!
I published it at my git server but here's the full source code, all 60 lines of it:
# Nicoletta, a minimal static site generator.
require "yaml"
require "markd"
require "crustache"
VERSION = "0.1.0"
# Load config file
tpl_data = File.open("conf") do |file|
YAML.parse(file).as_h
end
class Template
@text : String
@compiled : Crustache::Syntax::Template
def initialize(path)
@text = File.read(path)
@compiled = Crustache.parse(@text)
end
end
# Load templates
templates = {} of String => Template
Dir.glob("templates/*.tmpl").each do |path|
templates[path] = Template.new(path)
end
class Post
@metadata = {} of YAML::Any => YAML::Any
@text : String
@link : String
@html : String
def initialize(path)
contents = File.read(path)
metadata, @text = contents.split("\n\n", 2)
@metadata = YAML.parse(metadata).as_h
@link = path.split("/")[-1][0..-4] + ".html"
@html = Markd.to_html(@text)
end
def render(template)
Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html})
end
end
Dir.glob("posts/*.md").each do |path|
post = Post.new(path)
rendered_post = post.render templates["templates/post.tmpl"]
rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled,
tpl_data.merge({
"content" => rendered_post,
}))
File.open("output/#{post.@link}", "w") do |io|
io.puts rendered_page
end
end