Skip to main content

Ralsina.Me — Roberto Alsina's website

Booting with runit

I wrote, some time ago, an ar­ti­cle ex­plain­ing what the usu­al lin­ux boot­ing pro­cess, usu­al­ly called SYSVinit does. Sur­pris­ing­ly, that ar­ti­cle is one of the most pop­u­lars on my site. I sup­pose it's be­cause it's a com­mon tech­nol­o­gy, and the ex­ist­ing docs had some is­sues, be­ing ei­ther too tech­ni­cal or some­thing.

In any case, that page serves a pur­pose, and that's good. I hope this new ar­ti­cle serves an­oth­er pur­pose: show­ing that there can be bet­ter/­faster/nicer/­more man­age­able ways to boot a Lin­ux-­like sys­tem.

To do that, I will show a tool called runit and the con­cepts be­hind it.

The tool

Runit is a re­write of D.J. Bern­stein's dae­mon­tools, and the ba­sic con­cept is ser­vice su­per­vi­sion.

A su­per­vised ser­vice has a small process (the su­per­vi­sor), which knows how to start the ser­vice, how to clean up if it goes down, and whether the ad­min wants the ser­vice up or down.

This has some ob­vi­ous ad­van­tages. If the ap­pli­ca­tion hap­pens to crash, the su­per­vi­sor will restart it. The su­per­vi­sor is small enough that the odds of it crash­ing are very smal­l, and if it hap­pened, there are ways to man­age that, too.

The in­ter­face to the ser­vices is more uni­for­m, too. There is a stan­dard mech­a­nism to see the process sta­tus, get its PID and many things that, in the SYSV world, re­quire han­dling on the lev­el of the ser­vice's scrip­t. This makes the task of writ­ing good ser­vice scripts much eas­i­er. Con­sid­er the fol­low­ing ex­am­ple, for the GPM ser­vice. They are func­tion­al­ly equiv­a­len­t.


#!/bin/dash
# GPM in runit

exec 2>&1

. /etc/sysconfig/mouse
exec gpm -D -m $DEVICE -t $MOUSETYPE >/dev/null

#!/bin/bash
# GPM in SYSVinit, copied from Fedora, simplified to make it functionally
# equivalent to the runit version.

# source function library
. /etc/init.d/functions

RETVAL=0
start() {
    daemon gpm -m $DEVICE -t $MOUSETYPE $OPTIONS
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/gpm
}

stop() {
    echo -n "Shutting down console mouse services: "
    killproc gpm
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/gpm
}
case "$1" in
start)
        start
        ;;
stop)
        stop
        ;;
restart|reload)
        stop
        start
        ;;
condrestart)
        if [ -f /var/lock/subsys/gpm ]; then
            stop
            start
        fi
        ;;
status)
        status gpm
        RETVAL=$?
        ;;
*)
        echo $"Usage: $0 {start|stop|restart|condrestart|status}"
        exit 1
esac

Keep in mind that in the SYSVinit ver­sion, kill­proc and dae­mon are ac­tu­al­ly shell func­tion­s, de­fined in /etc/init.d/­func­tions, and they are rough­ly 70 lines each. Oh, and if gpm goes down, on Fe­do­ra it stays down.

So, I think we can agree runit can make some things sim­pler. Let's see how it work­s.

Anatomy of a service

A ser­vice in runit is not a scrip­t, but a di­rec­to­ry con­tain­ing some spe­cif­ic files in it.

run

In the most ba­sic se­tup, all you need is a run scrip­t. It's a reg­u­lar shell scrip­t, noth­ing strange about it at al­l. I nor­mal­ly use dash as shell for these things, be­cause it's small­er than bash, and so a lit­tle bit faster.

This script has to do a sim­ple task:

  1. per­­form any in­i­­tial­iza­­­tion the ser­vice re­quires (like sour­c­ing con­­fig files for var­i­ables, set­up path­s, what­ev­er)

  2. Start the ser­vice.

The ser­vice should not go in­to dae­mon mod­e, it should run "in the fore­ground". If you want to keep logs for this ser­vice, it is good if you can make it sim­ply log to stan­dard out­put or stan­dard er­ror.

You can see what op­tions work like this very sim­ply. Start the ser­vice in an in­ter­ac­tive shel­l. It should not give you back your promp­t, and it should print logs on the screen.

The exec 2>&1 on the GPM example simply makes the standard error stream go the standard output one, so it unifies both.

So, cre­ate that scrip­t, make it ex­e­cutable, and put it in­to a di­rec­to­ry.

finish

If there is a spe­cif­ic cleanup that is re­quired when­ev­er the ser­vice ex­it­s, cre­ate a fin­ish script in the same di­rec­to­ry. For ex­am­ple, sup­pose you want to send email to root when­ev­er the ser­vice ex­it­s, be­cause it's su­per-im­por­tan­t.


#!/bin/dash

# Finish script for important service!

echo `date` 'The important service crashed!' | mail -s 'crash!' admin@bigcorp.com

log

Logs are a very im­por­tant thing for an ad­min. They are your win­dow in­to the sys­tem. Runit in­cludes an ex­cel­lent mech­a­nism for log­ging, which is much sim­pler than sys­log. If there is a log sub­di­rec­to­ry in your ser­vice, it will treat it as a ser­vice, too, and will feed the stan­dard out­put of your ser­vice to the log's run scrip­t. That should be a script that logs its in­put some­where.

So, it looks like this:

ser­vice/run ==> ser­vice/log/run ==> log­file

Here's my uni­ver­sal log scrip­t:


#!/bin/dash

svname=servicename

if [ ! -d /var/log/$svname ]
then
        mkdir /var/log/$svname
        chown root.root /var/log/$svname
        chmod 700 /var/log/$svname
fi

exec /sbin/svlogd -tt /var/log/$svname

Re­place ser­vi­ce­name with what­ev­er you wan­t, save that as ser­vice/log/run, and that's it. (BTW, if some­one knows how to get the name of the par­ent fold­er in shell script­s, we can make that re­al­ly uni­ver­sal ;-)

Of course, the im­por­tant com­mand in the script is svlogd. It's part of runit. It's a tool that logs what­ev­er it gets from stan­dard in­put. The -tt op­tion makes it time­stamp in a hu­man-read­able for­mat.

Svlogd al­so man­ages au­to­mat­i­cal­ly log ro­ta­tion, which makes tools like logro­tate un­nec­es­sary. For that, read the doc­s, but here's what I do. I add a log/­con­fig file like this:


# Log a MB per-file, keep last ten files

s1000000
n10

No­tice that svlogd ro­tates logs by size, not by date. That's a mat­ter of ar­gu­ment in sysad­min cir­cles. You can al­so do stuff like bzip2-­com­press your old logs, us­ing the pro­ces­sor fea­ture, if you wan­t.

You could, if you wan­t, use oth­er same-style log­ging tool­s, like flog, mul­ti­log or many oth­er­s. You could even use the reg­u­lar sys­log.

Service control

When you have a ser­vice, you have to be able to start it, stop it, etc.

The first thing you have to do is start the ser­vice su­per­vi­sor, svrun. This will nor­mal­ly not be done man­u­al­ly, but by some­thing like runsvdir, which starts su­per­vi­sors for all ser­vices in a giv­en fold­er (see be­low for ex­am­ple). Once the su­per­vi­sor is run­ning, here's what you can do.

In the su­per­vised ser­vice paradig­m, the ser­vice has a state (run­ning/stopped) and a con­fig­ured state (how it wants to be, up­/­down).

So, a ser­vice can be:

  • Run­ning, down:

    It will not be restart­ed if it fin­ish­es.

  • Run­ning, up:

    It will be restart­ed if need­ed.

  • Stopped and up:

    It will be re­­tart­ed (prob­a­bly very soon).

  • Stopped and down:

    Will not start un­­less you do it man­u­al­­ly.

The tool to see the sta­tus is runsv­s­tat:

bash-2.05b# runsvstat /var/service/dropbear/
/var/service/dropbear/: run (pid 22756) 6 seconds

No­tice how it al­so gives ser­vice up­ti­me, which is a use­ful thing, usu­al­ly hard to get in SYSVinit.

The tool to con­trol a ser­vice is runsvc­trl:

#Switch it to up
runsvctrl u /var/service/dropbear

#Switch it to down
runsvctrl d /var/service/dropbear

#Make it run, but not switch to up (run once)
runsvctrl o /var/service/dropbear

It al­so has oth­er com­mand­s, like 'send KILL sig­nal' or 'send STOP sig­nal', so read the doc­s.

The runit-init boot

WARN­ING: DO NOT TRY TO SWITCH TO RUNIT ON AN IM­POR­TANT BOX UN­LESS YOU RE­AL­LY KNOW WHAT YOU ARE DO­ING

The main docs about boot­ing with runit can be found here but let me give you a short tour.

getties

Make sure you cre­ate a set of de­cent ser­vices for get­ties (ex­am­ple) or else you can't lo­gin.

Basic startup

Runit run­s, when start­ing the boot, the /etc/runit/1 scrip­t.

This should be the equiv­a­lent of Red Hat/Fe­do­ra's rc.sysinit, and you can fig­ure out how to do it by care­ful­ly fol­low­ing your sys­tem's boot up to when it goes in­to run­levels (see my ar­ti­cle if you must). The main goal of this script is run­ning ev­ery thing that has to run at boot and then nev­er again.

Here's an ex­am­ple for a ucrux based dis­tro I am hack­ing for my­self:


#!/bin/dash

PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin
D=`date`

echo "The system is coming up.  Please wait."

echo  Load configuration
. /etc/rc.conf

echo  Start device management daemon
/sbin/devfsd /dev

echo  Activate swap
/sbin/swapon -a

echo  Mount root read-only
/bin/mount -n -o remount,ro /

echo  Check filesystems
/sbin/fsck -A -T -C -a
if [ $? -gt 1 ]; then
        echo
        echo "***************  FILESYSTEM CHECK FAILED  ******************"
        echo "*                                                          *"
        echo "*  Please repair manually and reboot. Note that the root   *"
        echo "*  file system is currently mounted read-only. To remount  *"
        echo "*  it read-write type: mount -n -o remount,rw /            *"
        echo "*  When you exit the maintainance shell the system will    *"
        echo "*  reboot automatically.                                   *"
        echo "*                                                          *"
        echo "************************************************************"
        echo
        /sbin/sulogin -p
        echo "Automatic reboot in progress..."
        /bin/umount -a
        /bin/mount -n -o remount,ro /
        /sbin/reboot -f
        exit 0
fi
echo  Mount local filesystems
/bin/mount -n -o remount,rw /
/bin/rm -f /etc/mtab*
/bin/mount -a -O no_netdev

echo $D > /tmp/start

echo  Clean up misc files
: > /var/run/utmp
/bin/rm -rf /forcefsck /fastboot /etc/nologin /etc/shutdownpid \
            /var/run/*.pid /var/lock/* /tmp/{.*,*} &> /dev/null

rm -Rf /tmp/.ICE-unix
/bin/mkdir -m 1777 /tmp/.ICE-unix

echo  Set kernel variables
/sbin/sysctl -p > /dev/null

echo  Update shared library links
/sbin/ldconfig

echo Updating module deps
/sbin/depmod -a &

echo  Configure host name
if [ "$HOSTNAME" ]; then
        echo "hostname: $HOSTNAME"
        /bin/hostname $HOSTNAME
fi

echo  Load random seed
if [ -f /var/tmp/random-seed ]; then
        /bin/cat /var/tmp/random-seed > /dev/urandom
fi

echo  Configure system clock
if [ ! -f /etc/adjtime ]; then
        echo "0.0 0 0.0" > /etc/adjtime
fi
if [ "$TIMEZONE" ]; then
        /bin/ln -sf /usr/share/zoneinfo/$TIMEZONE /etc/localtime
fi
/sbin/hwclock --hctosys

echo  Start log daemons
/usr/sbin/syslogd
/usr/sbin/klogd -c 4

echo  Load console font
if [ "$FONT" ]; then
        echo "font: $FONT"
        /usr/bin/setfont $FONT
fi

echo  Load console keymap
if [ "$KEYMAP" ]; then
        echo "keyboard: $KEYMAP"
        /bin/loadkeys -q $KEYMAP
fi

echo  Screen blanks after 15 minutes idle time
/usr/bin/setterm -blank 15

echo Config hardware
/etc/rc.d/coldplug start

# End of file

The bad news are, you will al­most cer­tain­ly have to hack this your­self, un­til some­one de­cides to start mak­ing runit-based dis­tros ;-)

Then, af­ter that's done, you should have the filesys­tems mount­ed, the lo­cale set, the font load­ed, etc, etc.

So, now it runs /etc/runit/2. This is a much sim­pler scrip­t:


#!/bin/sh

PATH=/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin

date >/tmp/end
exec env - PATH=$PATH \
runsvdir /var/service 'log: ...................................................................................\
...............................................................................................................\
...............................................................................................................\
..........................................................................................'

This one on­ly as­sumes you cre­ate all your ser­vices in /var/ser­vice, so ex­cept for that, you can prob­a­bly sue it as-is.

What runsvdir does is to start all the ser­vices in /var/ser­vice. At the same time. This par­al­lel­ism will do won­ders for your boot time! It can al­so ren­der the sys­tem com­plete­ly un­bootable, so...

Service dependencies

Sup­pose you want to make sure gpm is runnng be­fore start­ing X, be­cause you are us­ing its re­peater mod­e. If you have no idea what I am talk­ing about, don't wor­ry. Just pre­tend that I gave you a valid rea­son for hav­ing a ser­vice de­pend on an­oth­er ;-)

Then, that means X's run script should make sure that gp­m's ser­vice is up be­fore ac­tu­al­ly start­ing X.

Here's how you do it:


#!/bin/dash
sv -w 3 start /var/service/gpm /var/service/getty* || exit 1
exec xdm -nodaemon

This script waits un­til gpm and ev­ery get­ty ser­vice I de­fined have been up for 3 sec­ond­s. If one of those ser­vices has a con­fig­ured state of down (mean­ing it will not start), then sv fails and the script ex­it­s.

The idea be­hind the 3 sec­onds is that you need some con­fi­dence that the ser­vices don't crash be­tween the sec­ond and third lines in our scrip­t. Sad­ly that is a race con­di­tion that can lead to trou­ble and is un­avoid­able. I haven't found a way to do it right, though, and in any case, SYSVinit does it worse.

The oth­er side of de­pen­den­cies is that if in our ex­am­ple gpm ex­it­s, X should al­so:


#!/bin/dash
sv stop -v -k -w 30 /var/service/X

That would be gp­m's fin­ish scrip­t. So, when gpm goes down, it takes X with it. Usu­al­ly, this is not a good idea, I am on­ly do­ing it for ex­am­ple's sake.

In the re­al world, most ser­vices will sur­vive grace­ful­ly, at least long enough so that the low­er-lev­el ser­vice restart­s. de­pen­den­cies on start­up are good. De­pen­den­cies on fin­ish are usu­al­ly dan­ger­ous and an­noy­ing.

I mean, killing X? What about all the stuff you are do­ing???? If GPM dies, at worst you are with­out a mouse. Well, at least you can save stuff us­ing short­cut­s, in­stead of be­ing dumped in­to a shell with­out warn­ing :-)

Not booting via runit

You can eas­i­ly boot us­ing SYSVinit yet use runit to man­age your ser­vices. Here's a sil­ly SYSVinit ser­vice script for runit:


#!/bin/sh

case "$1" in
start)
        runsvdir /var/service
        ;;
stop)
        killall runsvdir
        ;;
restart|reload)
        killall runsvdir
        runsvdir /var/service
        ;;
esac

So, use that as your on­ly SYSVinit ser­vice and be hap­py :-)

Runit, runit good!

So, it takes a cer­tain amount of work switch­ing to runit. You end up writ­ing or re­touch­ing many scripts (although there are a bunch of them in runit's site. What do you get for your ef­fort­s?

Well, you get a bet­ter sys­tem, I think. The main things I find are:

  • Bet­ter man­age­men­t:

    Runit ser­vices are more uni­­for­m, and bet­ter be­haved. You get more mean­ing­­ful sta­­tus in­­­for­­ma­­tion.

  • Bet­ter de­pen­den­cies:

    If a ser­vice de­pends on an­oth­er, you get to ac­­tu­al­­ly check to see if it work­s, in­­stead of trust­ing luck, which is what SYSVinit does, pret­­ty much.

  • Sim­pler ser­vice script­s:

    If you add your own ser­vices, SYSVinit is much hard­er. Since I do that of­ten, runit makes my life sim­­pler.

  • Faster boot­s:

    The start­up par­al­leliza­­­tion runit does is good. Usu­al­­ly your CPU is quite idle on star­­tup, while SYSV starts painful­­ly one ser­vice at a time. I have cut my boot time in half by switch­ing to runit in a Fe­­do­ra 3 test sys­tem, with no loss of func­­tion­al­i­­ty.

  • Less hack­ish:

    The con­­cept of a runit ser­vice is well de­fined. The con­­cept of log­ging is well de­fined. The con­­cept of star­­tup, of whether it should start on boot or not, are well de­fined, and in sim­­ple man­n­er­s, and it in­­­cludes the tools to man­age the ser­vices ef­fec­­tive­­ly. That means you don't have bolt­ed-on stuff like chk­­con­­fig, ntsysv, rc­­conf and so on. SYSVinit is used ev­ery­where, but the tools are dif­fer­­ent al­­most ev­ery­where.

And last but not least:

  • No run­level­s:

    The con­­cept of run­levels is evil. It serves no use­­ful pur­­pose, and makes SYSVinit much hard­er than it needs to be.

    But what the heck, if you want run­level­s, you can have them with runit as well.

If you try this, and are hap­py with the re­sult­s, (or even if you are un­hap­py ;-) please let me know!

If you are us­ing RHEL4 or Cen­tOS, then you can prob­a­bly avoid much trou­ble us­ing my runit RPM. It re­quires /bin/­dash which you can get from my dash RPM (or just link some shell to /bin/­dash).

After you install it, try booting with the init=/s­bin/runit-init parameter passed to your kernel and it should boot just fine.

Added May 19 2006: I post­ed this about con­vert­ing SysV scripts to runit.

Bill / 2011-03-16 21:18:

(BTW, if someone knows how to get the name of the parent folder in shell scripts, we can make that really universal)

This works in bash:
svname=`basename $(dirname $PWD)`

employment background check / 2011-12-27 23:21:


Hi very nice article


Contents © 2000-2020 Roberto Alsina