This is an oc­ca­sion­al se­ries of posts where I will share ideas for pro­grams that don't ex­ist, but should. The goal is to in­spire de­vel­op­ers to cre­ate use­ful tools that can make our lives eas­i­er. Or, more like­ly, to re­mind me about these ideas so I can cre­ate them my­self. Or even more like­ly, to just get them out of my head so I can stop think­ing about them.

Idea 3: A program that does backups the way I want

I don't want to con­fig­ure things on many com­put­er­s. I want to con­fig­ure things once in one com­put­er and have back­ups for ev­ery­thing I care about, done prop­er­ly.

There is a great back­up pro­gram called restic that does back­ups RIGHT. It is fast, it is se­cure, it is easy to use, and it has a lot of fea­tures.

There is a frontend for it called backrest that does a lot of things right. It separates the concept of repo which is what you backup to and plan which is what and when you backup.

But I want more, I want to separate plan (when and how) from source (where the data is)

I want to work like this:

1. Create a repo like "this folder here on this computer is a repo called 'foo' and it has a password and whatnot"
2. Create a plan like "every day at 3 AM, backup to 'foo', keep 30 days of backups there, and every week at 2am on sundays, backup to 'bar' which is a remote repo and keep 10 weeks of backups there"
3. Create a source like "the /home/ralsina folder in my notebook"
4. Create a backup which is a combination of a source and a plan like "backup the /home/ralsina folder in my notebook to 'foo' every day at 3 AM"

Fur­ther, I want this to just work on all my com­put­ers as long as I have ssh cor­rect­ly con­fig­ured to al­low it.

My back­up con­troller should log in­to what­ev­er com­put­er the re­po is in and in­stall restic there. Then log in­to the source com­put­er anb in­stall restic there. Then cre­ate the re­po, and when the plan says it's time to back­up, it should log in­to the source com­put­er and run the back­up com­mand there, and then log in­to the re­po com­put­er and run some­thing there if it needs run­ning.

Then I want it to keep logs and no­ti­fy me via go­ti­fy or some­thing if some­thing goes wrong.

Is that too much to ask?

This is a new oc­ca­sion­al se­ries of posts where I will share ideas for pro­grams that don't ex­ist, but should. The goal is to in­spire de­vel­op­ers to cre­ate use­ful tools that can make our lives eas­i­er. Or, more like­ly, to re­mind me about these ideas so I can cre­ate them my­self. Or even more like­ly, to just get them out of my head so I can stop think­ing about them.

Idea 2: A nice web frontend for journald

I do some self host­ing. It's tempt­ing, when you self­-host, to run things as if it was a com­pa­ny's pro­duc­tion set­up. So, there are some who run mul­ti­ple large servers on ku­ber­netes and so on.

Not me, I run a sin­gle SBC with a bunch of dock­er­ized ser­vices.

So, how do I see logs if some­thing goes wrong?

Well, I log to the system's journal, so I can use journalctl to see the logs.

It's just this bit of YAML in your com­pos­er def­i­ni­tion:

  logging:
    driver: "journald"
    options:
      tag: "whatever"

That tags the logs from that con­tain­er with "what­ev­er". So, I can run:

journalctl -t whatever

This tool, journalctl is quite nice, and you can filter by date, grep for things, follow the live logs, and so on. But it's a command line tool, which I like.

But the ¨do it like a re­al prod thing"crowd us­es logstash or some­such, and have a web dash­board for this kind of things.

Well, I should have one of those too, but backed by journalctl

There is one that comes with sys­temd, but it's sort of crap­py, and there is no rea­son for it to be. It's run­ning in the same serv­er where the logs are, it's sim­ple, and it would be a nice lit­tle project to do.

This is a new oc­ca­sion­al se­ries of posts where I will share ideas for pro­grams that don't ex­ist, but should. The goal is to in­spire de­vel­op­ers to cre­ate use­ful tools that can make our lives eas­i­er. Or, more like­ly, to re­mind me about these ideas so I can cre­ate them my­self. Or even more like­ly, to just get them out of my head so I can stop think­ing about them.

Idea 1: A program that can automatically configure complex machine/monitor setups

My desk­top has mul­ti­ple ma­chines shar­ing mul­ti­ple mon­i­tors via one or more HD­MI switch­es. Those mon­i­tors are land­scape or por­trait, some are built in­to lap­tops and oth­ers are ex­ter­nal. Some are di­rect­ly con­nect­ed, some are not. A com­put­er may have be­tween ze­ro and three mon­i­tors con­nect­ed at any one time.

When they are con­nect­ed, in some cas­es, I want to run bar­ri­er to share the mouse and key­board, but with dif­fer­ent con­fig­u­ra­tions de­pend­ing on what mon­i­tors are con­nect­ed to what (say, is this com­put­er us­ing the mon­i­tor at the left of the one for the com­put­er with the key­board and mouse?).

So:

• I want to be able to con­fig­ure the mon­i­tors and their ori­en­ta­tion­s, and the key­board­/­mouse shar­ing, for each ma­chine.
• This should hap­pen au­to­mat­i­cal­ly when I con­nect a mon­i­tor or switch the HD­MI switch.
• It should start/stop/re­con­fig­ure bar­ri­er on each ma­chine.

Of course this does­n't ex­ist and is pretttt­ty hard to do.

For de­tails on how my set­up works now: mon­i­tors and kb­d/­mouse

I have a few small projects where I need a serv­er as back­end. That means I need to run a server, which usu­al­ly means I need to do a lot of stuff. BUT these are as­ton­ish­ing­ly sim­ple back­end­s. Usu­al­ly just one end­point, which does one thing.

For ex­am­ple, con­sid­er nom­bres a web­site where you can ex­am­ine his­tor­i­cal in­for­ma­tion about names in Ar­genti­na. Like, how has the pop­u­lar­i­ty of the name "Juan" changed over time?

Like this:

That is lit­er­al­ly one func­tion that takes as ar­gu­ment names, does a cou­ple of queries to a database, builds a chart and re­turns that. De­ploy­ing that should not re­quire me set­ting up in­fra­struc­ture spe­cial­ly be­cause I have like 5 or 10 of those and they are ac­cessed 10 times a day or so.

If I were to use "the cloud" the so­lu­tion would be to use AWS Lamb­da, or the sim­i­lar clones in Azure or Google Cloud. But I don't want to pay for things, so I looked for a way to do that in my own server, which I al­ready have and has more than enough pow­er to han­dle it.

So, I did it! I used faasd which lead to a whole slew of prob­lems that you can see in this post.

Ba­si­cal­ly faasd hates shar­ing the ma­chine it's in with any­thing else, so I seg­re­gat­ed it to a VM us­ing Ig­nite. Now, I am con­sid­er­ing mov­ing to a new­er, more pow­er­ful serv­er (A Radxa Rock 5C) and I look at set­ting up Ig­nite and ... it's ob­so­lete.

It says the re­place­ment is Flint­lock­... which is "on hold", which means aban­doned.

So I try set­ting up a VM us­ing lib­virt, but since this is ar­m64, things are a bit com­pli­cat­ed, so I say, damn, let's just use QE­mu, which works ... as long as I don't use KVM, be­cause it makes the VM su­per flaky.

So I can choose a su­per flaky VM, or a very slow VM, or use ob­so­lete soft­ware, all be­cause I want to run a func­tion that takes a few sec­onds to run and is ac­cessed 10 times a day and faasd is needy.

So, I say­d, what the heck, faasd can't be the on­ly thing. Let's look again!

• Open­Faas: by the same peo­ple as faas­d, needs ku­ber­netes, way overkill
• Fx: looks good and sim­ple, lit­er­al­ly does­n't work.

You don't be­lieve me? Here is what hap­pens when I fol­low the in­struc­tions to run fx, on a nor­mal x86 ma­chine:

> curl -o- https://raw.githubusercontent.com/metrue/fx/master/scripts/install.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1471  100  1471    0     0   4138      0 --:--:-- --:--:-- --:--:--  4143
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8635  100  8635    0     0  17948      0 --:--:-- --:--:-- --:--:-- 17914
Downloading fx from https://github.com/metrue/fx/releases/download/0.9.48-alpha.d91a7a0/fx_0.9.48-alpha.d91a7a0_Tux_64-
Download complete, saved to /home/ralsina/fx/fx.tar.gz
Installing fx to /home/ralsina/fx
fx
fx installed successfully at /home/ralsina/fx
fx version 0.9.48
Cleaning up /home/ralsina/fx/fx.tar.gz

> cat > func.js
module.exports = (ctx) => {
  ctx.body = 'hello world'
}

> ./fx up -p 8080 func.js
2024/06/28 10:52:54  info provisioning localhost ...
*****************
exit status 125
*****************

I spent an hour or so on it, it just does­n't work.

What else is out there?

• Apache Open­Whisk looks promis­ing, I will try it out nex­t, need to see if it works in my re­source-lim­it­ed serv­er.

If it does­n't work out I swear I am rolling my own.

UP­DATE: Looks like I am rolling my own:

I have a server, her name is Pinky

Pinky does a lot of things but pinky has one prob­lem: Pinky is to­tal­ly hand-­made. Ev­ery­thing in it has been in­stalled by hand, con­fig­ured by hand, and main­tained by hand. This is ok.

I mean, it's ok, un­til it's not ok. It has back­ups and ev­ery­thing, but when a chance presents to, for ex­am­ple, move to a new server, be­cause I just got a nice new com­put­er ... I would need to do ev­ery­thing by hand again.

So, let's fix this us­ing tech­nol­o­gy. I have known about an­si­ble for a long time, I have used things like an­si­ble. I have used pack­er, and salt, and pup­pet, and (re­lat­ed) dock­er, and ku­ber­netes, and ter­rafor­m, and cloud­for­ma­tion, and chef, and ... you get the idea.

But I have nev­er used an­si­ble!

So, here's my plan:

• I will start do­ing an­si­ble play­books for pinky.
• Since an­si­ble is idem­po­ten­t, I can run the play­books on pinky and noth­ing should change.
• I can al­so run them on the new server, and ev­ery­thing should be set up.
• At some point the new serv­er will be suf­fi­cient­ly pinky-­like and I can switch.

So, what is ansible?

In non-tech­ni­cal terms: An­si­ble is a tool to change things on ma­chines. An­si­ble can:

• Set­up a us­er
• Copy a file
• In­stall a pack­age
• Con­fig­ure a thing
• En­able a ser­vice
• Run a com­mand

And so on.

Ad­di­tion­al­ly:

• It will on­ly do things that need to be done.
• It will do things in the re­quest­ed or­der.
• It will do things in mul­ti­ple ma­chines.

First: inventory

The first thing I need to do is to tell an­si­ble where to run things. This is done us­ing an in­ven­to­ry file. The in­ven­to­ry file is a list of ma­chi­nes, and groups of ma­chi­nes, that an­si­ble can run things on.

Mine is very sim­ple, a file called hosts in the same di­rec­to­ry as the play­book:

[servers]
pinky ansible_user=ralsina
rocky ansible_user=rock

[servers:vars]
ansible_connection=ssh 

This defines two machines, called pinky (current server) and rocky (new server). Since rocky is still in pretty much brand new shape it has only the default user it came with, called rock. I have logged into it and done some things ansible needs:

• En­abled ssh
• Made it so my per­son­al ma­chine where an­si­ble runs can log in with­out a pass­word
• In­stalled python
• Made rock a sudoer so it can run commands as root using sudo

So, I tell ansible I can log in as ralsina in pinky and as rock in rocky, in both cases using ssh.

First playbook

I want to be able to log into these machines using my user ralsina and my ssh key. So, I will create a playbook that does that. Additionally, I want my shell fish and my prompt starship to be installed and enabled.

A play­book is just a YAML file that lists tasks to be done. We start with some gener­ic stuff like "what ma­chines to run this on" and "how do I be­come root?"

# Setup my user with some QoL packages and settings
- name: Basic Setup
  hosts: servers
  become_method: ansible.builtin.sudo
  tasks:

And then guess what? Tasks. Each task is a thing to do. Here's the first one:

    - name: Install some packages
      become: true
      ansible.builtin.package:
        name:
          - git
          - vim
          - htop
          - fish
          - rsync
          - restic
          - vim
        state: present

There "an­si­ble.builtin.­pack­age" is a mod­ule that in­stalls pack­ages. An­si­ble has tons of mod­ules, and they are all doc­u­ment­ed in the an­si­ble doc­u­men­ta­tion.

Each task can take parameters, which depend on what the module does. In this case, as you can see there's a list of packages to install, and the state means I want them to be there.

BUT while rocky is a Debian, pinky is arch (btw), so there is at least one package I need to install only in rocky. That's the next task:

    - name: Install Debian-specific packages
      become: true
      when: ansible_os_family == 'Debian'
      ansible.builtin.apt:
        name:
          - ncurses-term
        state: present

Same thing, ex­cep­t:

• It uses a debian-specific package thing, called ansible.builtin.apt
• It has a when clause that only runs the task if the OS family is Debian.

What nex­t? Well, more tasks! Here they are, you can un­der­stand what each one does by look­ing up the docs for each an­si­ble mod­ule.

    - name: Add the user ralsina
      become: true
      ansible.builtin.user:
        name: ralsina
        create_home: true
        password_lock: true
        shell: /usr/bin/fish
    - name: Authorize ssh
      become: true
      ansible.posix.authorized_key:
        user: ralsina
        state: present
        key: "{{ lookup('file', '/home/ralsina/.ssh/id_rsa.pub') }}"
    - name: Make ralsina a sudoer
      become: true
      community.general.sudoers:
        name: ralsina
        user: ralsina
        state: present
        commands: ALL
        nopassword: true
    - name: Create fish config directory
      ansible.builtin.file:
        path: /home/ralsina/.config/fish/conf.d
        recurse: true
        state: directory
        mode: '0755'
    - name: Get starship installer
      ansible.builtin.get_url:
        url: https://starship.rs/install.sh
        dest: /tmp/starship.sh
        mode: '0755'
    - name: Install starship
      become: true
      ansible.builtin.command:
        cmd: sh /tmp/starship.sh -y
        creates: /usr/local/bin/starship
    - name: Enable starship
      ansible.builtin.copy:
        dest: /home/ralsina/.config/fish/conf.d/starship.fish
        mode: '0644'
        content: |
          starship init fish | source

And that's it! I can run this playbook using ansible-playbook -i hosts setup_user.yml and it will do all those things on both pinky and rocky, if needed:

> ansible-playbook -i hosts setup_user.yml

PLAY [Basic Setup] ******************************

TASK [Gathering Facts] **************************
ok: [rocky]
ok: [pinky]

TASK [Install some packages] ********************
ok: [rocky]
ok: [pinky]

TASK [Install Debian-specific packages] *********
skipping: [pinky]
ok: [rocky]

TASK [Add the user ralsina] *********************
ok: [rocky]
ok: [pinky]

TASK [Authorize ssh] ****************************
ok: [rocky]
ok: [pinky]

TASK [Make ralsina a sudoer] ********************
ok: [rocky]
ok: [pinky]

TASK [Create fish config directory] *************
changed: [rocky]
changed: [pinky]

TASK [Get starship installer] *******************
ok: [rocky]
ok: [pinky]

TASK [Install starship] *************************
ok: [rocky]
ok: [pinky]

TASK [Enable starship] **************************
changed: [rocky]
changed: [pinky]

PLAY RECAP **************************************
pinky : ok=9    changed=2    unreachable=0    failed=0    skipped=1 
        rescued=0    ignored=0
rocky : ok=10   changed=2    unreachable=0    failed=0    skipped=0 
        rescued=0    ignored=0

If you look care­ful­ly you can see rocky ran one more task, and pinky skipped one (the de­bian-spe­cif­ic pack­age in­stal­la­tion), and that on­ly two things got ac­tu­al­ly ex­e­cut­ed on each ma­chine.

I could run this a dozen times from now on, and it would not do any­thing.

Did it work?

Sure, I can ssh into rocky and everything is nice:

> ssh rocky
Linux rock-5c 5.10.110-37-rockchip #27a257394 SMP Thu May 23 02:38:59 UTC 2024 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Jun 26 15:32:33 2024 from 100.73.196.129
Welcome to fish, the friendly interactive shell
Type `help` for instructions on how to use fish

ralsina in 🌐 rock-5c in ~ 

There is a star­ship promp­t, and I can use fish. And I can su­do. Nice!

I can now change the inventory so rocky also uses the ralsina user and delete the rock user.

Next steps

There is a lot more to an­si­ble, specif­i­cal­ly roles but this is al­ready enough to get use­ful things done, and hope­ful­ly it will be use­ful to you too.