Skip to main content

Ralsina.Me — Roberto Alsina's website

Making Linux systems that don't suck. Part I

I wrote in a pre­vi­ous post that there is a whole lot of soft­ware shipped with most lin­ux dis­tros which sucks (although I said a it a bit more mild­ly).

You could con­sid­er an old post of mine, called Log­ging in style as the 0th part of this se­ries. Please read it first. It's prob­a­bly bet­ter than this one.

Now, I will tack­le an­oth­er, re­lat­ed area... How does your sys­tem tell you what hap­pened?

You may say (spe­cial­ly if you read the piece I men­tioned) "Through the sys­tem logs!". And you are right. Ex­cept that do­ing it that way is crazy.

Mind you, logs are not crazy, it's re­ly­ing on them to know what hap­pens that's crazy.

Logs are great to know what hap­pened when you know what you are look­ing for. Not a way to know what hap­pened that was im­por­tan­t.

Right now, the tool your Lin­ux box us­es to tell you what hap­pened is prob­a­bly send­mail [1].

Here are a few ex­am­ples:

  • Ev­ery pro­­grammed task us­ing at or cron mails its out­­put to the ad­min­is­­tra­­tor. If you do any sysad­min work you know this is very im­­por­­tan­t.

  • Log­watch is run by cron, ex­am­ines your logs and gives you a re­­port on sys­tem ac­­tiv­i­­ty. And mails it.

  • Bac­u­la (my favourite se­ri­ous back­­up tool) runs its back­­up and re­­s­tore job­s, then mails the out­­put to the back­­up ad­min­is­­tra­­tors.

  • Munin or Ar­­gus mon­i­­tor sys­tems and ser­vices on many com­put­er­s, and mail you if things go wrong.

And so on, and so forth. Al­most ev­ery tool that needs to alert you if some­thing hap­pened, or needs to re­port to you the re­sults of a sched­uled task, will end up mail­ing you.

This is spe­cial­ly an­noy­ing in that the de­fault con­fig­u­ra­tion in all Lin­ux Dis­tros is to do this kind of things in a way that can't pos­si­bly be what you wan­t: Mail­ing this stuff to the lo­cal root ac­count.

Let's con­sid­er some of the ways in which this is wrong...

  1. Usu­al­­ly you can't read root's mail us­ing POP or IMAP (and you re­al­­ly should­n't) [2].

  2. So you need to read it lo­­cal­­ly. Us­ing what? mail(1)? mut­t? Or are you ask­ing the us­er to lo­­gin at the GUI as root? At least for­ward root's mail to the us­er ac­­count you cre­ate on in­­stal­la­­tion, please! (no idea if any dis­­tro does this).

  3. If you nev­er both­­er read­­ing and delet­ing those mail­s, they will even­­tu­al­­ly fill your / or /home par­ti­­tion (if you use ~/­­Maildir as mail store) or your /var par­ti­­tion (for /var/spool/­­mail store).

Sup­pose you know this al­ready. You fix it. [3] You for­ward root's mail to an ac­count you ac­tu­al­ly read. Usu­al­ly that should mean a re­mote ac­coun­t, of course.

Ig­nor­ing the fact that mail de­liv­ery is get­ting less and less re­li­able ev­ery day (at least don't use a hot­mail ac­coun­t, ok?) let's as­sume you are get­ting all this ad­min­is­tra­tive mail.

What's the prob­lem now?

That you are run­ning a gen­er­al pur­pose mail serv­er. On ev­ery box. All the time. Prob­a­bly al­low­ing mail for ev­ery ac­count in the sys­tem, and usu­al­ly open­ing a port, even if on­ly in the loop­back in­ter­face.

Does that make any sense? Why do you have a ser­vice so com­plex and an­noy­ing in­stalled? Why use a soft­ware you can buy books about? Is it nec­es­sary? I say no. We'll see if I'm right.

Now, back to the re­li­a­bil­i­ty of mail is­sue: You need to con­fig­ure a mail serv­er right, or use a smart host as a mail gate­way, or you are go­ing to lose mail.

If you are go­ing to ig­nore ev­ery­thing else in this post, don't ig­nore this: use a smart host. Please!

We have enough mal­formed mail float­ing around. We don't need an­oth­er mil­lion mes­sages from root@lo­cal­host to ad­min2354@g­mail.­com.

If you can, use your own do­main's mail serv­er as smarthost for your box­es, so noone needs to for­ward your mail.

If you can't do that, hire a vir­tu­al box from tek­ton­ic, set up a re­ceive-on­ly mail serv­er with au­then­ti­cat­ed SMTPS and/or ac­cess lim­it­ed to spe­cif­ic fixed IP ad­dress­es, and use that as a smart host so your ad­min­is­tra­tive mail does­n't float around caus­ing trou­bles for us on­look­er­s.

Now let's get con­struc­tive. What can we do about it?

The first good thing is that there is re­al­ly no rea­son why all these no­ti­fi­ca­tions need to be sent via email.

What's hap­pen­ing is that all these dif­fer­ent ap­pli­ca­tions are prob­a­bly us­ing send­mail. But that's just send­mail the com­mand, not send­mail the MTA or any oth­er MTA. The use of the MTA is just a side ef­fec­t.

So, we have a sim­ple path ahead of us:

  1. De­­cide how you want to re­­ceive your no­ti­­fi­­ca­­tion­s. Store tham in a DB? Send you IM mes­sages? Read them aloud? Store files in a fold­er struc­­ture? Show pop­up di­alogs?

  2. Im­­ple­­ment a send­­mail in­­ter­­face that does ex­ac­t­­ly that.

I should prob­a­bly ad­d: does ex­act­ly that in the sim­plest way pos­si­ble.

I have not yet thought through step 1, though. So, I am still us­ing email. But for that spe­cif­ic case, I have step 2 fig­ured out: null­mail­er.

  • Usu­al­­ly you need to ed­it on­­ly two lines to make it work nice­­ly with a smarthost (a work­ing con­­fig­u­ra­­tion is prob­a­bly about 4 lines).

  • Its size is a whoop­ing 227KB in my box.

  • The de­­tailed docs are about 300 ter­mi­­nal lines long (The HOW­­TO is 54 lines).

  • It us­es al­­most no re­­sources what­­so­ev­er when not send­ing mail: about 1MB of ram (450KB RSS) and 1 CPU-sec­ond a day on my note­­book.

  • It will prob­a­bly force you to use runit or dae­­mon­­tools and a de­­cent log­ger even if it's on­­ly for this one ser­vice ;-)

Why are no dis­tros ship­ping this or some­thing like it? Do this, add a triv­ial con­fig­u­ra­tion tool, and your com­put­er is sud­den­ly 10 times bet­ter cared for.

If I were man­ag­ing a com­mer­cial dis­tro, I would give mail ac­counts to all my pay­ing cus­tomers and ship all box­es with a de­fault set­ting of "put the re­ports on this web­page" (to be en­abled by the user). It would cost al­most noth­ing, and pre­vent part of the "I nev­er knew 25 peo­ple logged via SSH on my root ac­count yes­ter­day" syn­drome.

And I would re­al­ly think hard about what a good de­liv­ery mech­a­nism is for this kind of mes­sages, and im­ple­ment it. Be­cause right now... it suck­s. Come on, IB­M/RH/Ubun­tu/Who­ev­er, this is not hard stuff, this is low hang­ing fruit. Show some gump­tion and pick it up!

Join me next week when I will ex­plain why cron and at suck.


Contents © 2000-2023 Roberto Alsina