Making Linux systems that don’t suck. Part I

I wrote in a pre­vi­ous post that there is a whole lot of soft­ware shipped with most lin­ux dis­tros which sucks (although I said a it a bit more mild­ly).

You could con­sid­er an old post of mine, called Log­ging in style as the 0th part of this se­ries. Please read it first. It’s prob­a­bly bet­ter than this one.

Now, I will tack­le an­oth­er, re­lat­ed area… How does your sys­tem tell you what hap­pened?

You may say (spe­cial­ly if you read the piece I men­tioned) “Through the sys­tem logs!”. And you are right. Ex­cept that do­ing it that way is crazy.

Mind you, logs are not crazy, it’s re­ly­ing on them to know what hap­pens that’s crazy.

Logs are great to know what hap­pened when you know what you are look­ing for. Not a way to know what hap­pened that was im­por­tan­t.

Right now, the tool your Lin­ux box us­es to tell you what hap­pened is prob­a­bly send­mail [1].

Here are a few ex­am­ples:

  • Ev­ery pro­grammed task us­ing at or cron mails its out­put to the ad­min­is­tra­tor. If you do any sysad­min work you know this is very im­por­tan­t.
  • Log­watch is run by cron, ex­am­ines your logs and gives you a re­port on sys­tem ac­tiv­i­ty. And mails it.
  • Bac­u­la (my favourite se­ri­ous back­up tool) runs its back­up and re­store job­s, then mails the out­put to the back­up ad­min­is­tra­tors.
  • Munin or Ar­gus mon­i­tor sys­tems and ser­vices on many com­put­er­s, and mail you if things go wrong.

And so on, and so forth. Al­most ev­ery tool that needs to alert you if some­thing hap­pened, or needs to re­port to you the re­sults of a sched­uled task, will end up mail­ing you.

This is spe­cial­ly an­noy­ing in that the de­fault con­fig­u­ra­tion in all Lin­ux Dis­tros is to do this kind of things in a way that can’t pos­si­bly be what you wan­t: Mail­ing this stuff to the lo­cal root ac­count.

Let’s con­sid­er some of the ways in which this is wrong…

  1. Usu­al­ly you can’t read root’s mail us­ing POP or IMAP (and you re­al­ly should­n’t) [2].
  2. So you need to read it lo­cal­ly. Us­ing what? mail(1)? mut­t? Or are you ask­ing the us­er to lo­gin at the GUI as root? At least for­ward root’s mail to the us­er ac­count you cre­ate on in­stal­la­tion, please! (no idea if any dis­tro does this).
  3. If you nev­er both­er read­ing and delet­ing those mail­s, they will even­tu­al­ly fill your / or /home par­ti­tion (if you use ~/­Maildir as mail store) or your /var par­ti­tion (for /var/spool/­mail store).

Sup­pose you know this al­ready. You fix it. [3] You for­ward root’s mail to an ac­count you ac­tu­al­ly read. Usu­al­ly that should mean a re­mote ac­coun­t, of course.

Ig­nor­ing the fact that mail de­liv­ery is get­ting less and less re­li­able ev­ery day (at least don’t use a hot­mail ac­coun­t, ok?) let’s as­sume you are get­ting all this ad­min­is­tra­tive mail.

What’s the prob­lem now?

That you are run­ning a gen­er­al pur­pose mail serv­er. On ev­ery box. All the time. Prob­a­bly al­low­ing mail for ev­ery ac­count in the sys­tem, and usu­al­ly open­ing a port, even if on­ly in the loop­back in­ter­face.

Does that make any sense? Why do you have a ser­vice so com­plex and an­noy­ing in­stalled? Why use a soft­ware you can buy books about? Is it nec­es­sary? I say no. We’ll see if I’m right.

Now, back to the re­li­a­bil­i­ty of mail is­sue: You need to con­fig­ure a mail serv­er right, or use a smart host as a mail gate­way, or you are go­ing to lose mail.

If you are go­ing to ig­nore ev­ery­thing else in this post, don’t ig­nore this: use a smart host. Please!

We have enough mal­formed mail float­ing around. We don’t need an­oth­er mil­lion mes­sages from root@lo­cal­host to ad­min2354@g­mail.­com.

If you can, use your own do­main’s mail serv­er as smarthost for your box­es, so noone needs to for­ward your mail.

If you can’t do that, hire a vir­tu­al box from tek­ton­ic, set up a re­ceive-on­ly mail serv­er with au­then­ti­cat­ed SMTPS and/or ac­cess lim­it­ed to spe­cif­ic fixed IP ad­dress­es, and use that as a smart host so your ad­min­is­tra­tive mail does­n’t float around caus­ing trou­bles for us on­look­er­s.

Now let’s get con­struc­tive. What can we do about it?

The first good thing is that there is re­al­ly no rea­son why all these no­ti­fi­ca­tions need to be sent via email.

What’s hap­pen­ing is that all these dif­fer­ent ap­pli­ca­tions are prob­a­bly us­ing send­mail. But that’s just send­mail the com­mand, not send­mail the MTA or any oth­er MTA. The use of the MTA is just a side ef­fec­t.

So, we have a sim­ple path ahead of us:

  1. De­cide how you want to re­ceive your no­ti­fi­ca­tion­s. Store tham in a DB? Send you IM mes­sages? Read them aloud? Store files in a fold­er struc­ture? Show pop­up di­alogs?
  2. Im­ple­ment a send­mail in­ter­face that does ex­act­ly that.

I should prob­a­bly ad­d: does ex­act­ly that in the sim­plest way pos­si­ble.

I have not yet thought through step 1, though. So, I am still us­ing email. But for that spe­cif­ic case, I have step 2 fig­ured out: null­mail­er.

  • Usu­al­ly you need to ed­it on­ly two lines to make it work nice­ly with a smarthost (a work­ing con­fig­u­ra­tion is prob­a­bly about 4 lines).
  • Its size is a whoop­ing 227KB in my box.
  • The de­tailed docs are about 300 ter­mi­nal lines long (The HOW­TO is 54 lines).
  • It us­es al­most no re­sources what­so­ev­er when not send­ing mail: about 1MB of ram (450KB RSS) and 1 CPU-sec­ond a day on my note­book.
  • It will prob­a­bly force you to use runit or dae­mon­tools and a de­cent log­ger even if it’s on­ly for this one ser­vice ;-)

Why are no dis­tros ship­ping this or some­thing like it? Do this, add a triv­ial con­fig­u­ra­tion tool, and your com­put­er is sud­den­ly 10 times bet­ter cared for.

If I were man­ag­ing a com­mer­cial dis­tro, I would give mail ac­counts to all my pay­ing cus­tomers and ship all box­es with a de­fault set­ting of “put the re­ports on this web­page” (to be en­abled by the user). It would cost al­most noth­ing, and pre­vent part of the “I nev­er knew 25 peo­ple logged via SSH on my root ac­count yes­ter­day” syn­drome.

And I would re­al­ly think hard about what a good de­liv­ery mech­a­nism is for this kind of mes­sages, and im­ple­ment it. Be­cause right now… it suck­s. Come on, IB­M/RH/Ubun­tu/Who­ev­er, this is not hard stuff, this is low hang­ing fruit. Show some gump­tion and pick it up!

Join me next week when I will ex­plain why cron and at suck.

[1] This is true at least on RHEL4 and several other “modern” Linuxes. But even if it’s Postfix or Exim: it’s wrong anyway.
[2] And if you misconfigure your POP/IMAP server so you can do it, you just hacked your own box.
[3] It would be better if you didn’t have to unbreak the system, but let’s ignore that for a while.


Comments powered by Disqus
Contents © 2000-2013 Roberto Alsina