Logging in style
Let's try a little classical logic:
Logs are the administrator's best friend.
Linux is the favourite operating system for many administrators.
Ergo, Linux has a very nice logging system.
BZZZZZT Linux has an adequate logging system. At least compared to what you get with some other operating systems, where often you have no place to look for the reason of your misery.
On Linux, your misery is usually visible :-)
How You Are Probably Doing It Now
You are very likely to be running syslogd, which is a pretty traditional unixy syslog implementation, or syslog-ng, which is the same thing, only different. We'll assume you are using syslog from now on.
It's configured via /etc/syslog.conf, which usually looks like this:
# Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log
To make a long story short: the left column is facility.priority The right column is where you log it.
For example, a critical message from your mail server software, will probably be mail.critical and be logged to /var/log/maillog.
A single message can be logged in as many places as you want.
The applications decide what facility they use and what priority their messages are.
Simple, right? Yes. But not very good.
Rotation
Suppose I know that whenever I connect to your mail server, you log a line, about 60 bytes.
Now, suppose I connect to your server 10 times per second for three days. That's 150MB of logs.
Now suppose my 20 best friends do it, too. That's 3GB of logs.
Basically, with some effort, anyone can clog your disk.
Oh, but your logs rotate I hear. Log rotation means you cut your logs every once in a while, delete older data, maybe compress not-so-old data, and that way the logs don't grow indefinitely. You probably use logrotate for this.
But just how large will your log get? Consider that the default rotation period is usually weekly, and usually you keep 4 older copies... but it doesn't matter: if you rotate based on time, you have no upper bound to the size of your logs.
It would be better to rotate by size of the log, or a combination of both things.
Who Manages Logs?
I usually manage the whole system. However, I think it's a good idea if you have the possibility of delegating specific subsystems to other admins.
Now suppose you are my sub-admin in charge of service FOO.
Now, you want to debug something, so you need to change FOO's logging to catch debug priority messages.
How do you do that? Well, you have to call me, because there is a centralized log configuration. (Unless FOO simply ignores system logging, which sucks in a completely different way).
Basically, we are not putting the management of logging for FOO close to FOO. We are putting it close to syslog. Maybe that's good. Maybe not.
Where's the log?
Unless you have a clear idea of what facility and priority your app uses, how do you know where it's logging? What happens if you only catch part of the log?
It would be nicer if each service had a specific place where it logged! Uniform is good.
What happens if syslog dies?
Well, you don't log anymore. So please make it a managed service, because life without logs sucks.
So, what can we use?
Well, as long as you use something like runit or daemontools, you can use one of the bazillion logging tools that use the "give me a folder name and data on stdin" model. I am partial to svlogd but there are a million others.
That solves many things:
There is no syslog, so it can't die. There is one logger for each service. And it's managed anyway, so it bounces back up.
The log is wherever you tell it to be. Standardize in /var/log/FOO or somesuch.
You can choose under what user the logger runs. So, whoever you want manages it.
Most loggers offer you size or time-based rotation. And you don't need to configure logrotate (you don't need logrotate)
But what about the apps that want to use syslogd and don't work as a managed service with logging?
Well, they do it by writing to /dev/log, a pseudo-device which pipes the data to syslog.
Then, you can use socklog a lovely drop-in drop-dead gorgeous replacement for syslogd. Use it, configure it for unix socket (using socklog-conf), start it (using runit's runsv), and enjoy:
[root@myhost socklog]# pwd /var/log/socklog [root@myhost socklog]# ls -lR .: total 48 drwxr-x--- 2 root root 4096 Sep 14 21:16 auth drwxr-x--- 2 root root 4096 Sep 14 21:16 cron drwxr-x--- 2 root root 4096 Sep 14 21:16 daemon drwxr-x--- 2 root root 4096 Sep 14 21:16 debug drwxr-x--- 2 root root 4096 Sep 14 21:16 ftp drwxr-x--- 2 root root 4096 Sep 14 21:16 kern drwxr-x--- 2 root root 4096 Sep 14 21:16 local drwxr-x--- 2 root root 4096 Sep 14 21:16 mail drwxr-x--- 2 root root 4096 Sep 14 21:16 main drwxr-x--- 2 root root 4096 Sep 14 21:16 news drwxr-x--- 2 root root 4096 Sep 14 21:16 syslog drwxr-x--- 2 root root 4096 Sep 14 21:16 user ./auth: total 8 -rw-r--r-- 1 root root 34 Sep 14 21:16 config -rw-r--r-- 1 root root 1003 Sep 14 22:03 current -rw------- 1 root root 0 Sep 14 21:16 lock ./cron: total 4 -rw-r--r-- 1 root root 22 Sep 14 21:16 config -rw-r--r-- 1 root root 0 Sep 14 21:16 current -rw------- 1 root root 0 Sep 14 21:16 lock
One folder for each facility, you can filter the data if you want to avoid logging specific stuff, timestamp any way you want, rotate at will, yada yada yada.
On Arch Linux, you can try this by installing my runit and socklog packages from AUR.
It should be pretty painless, but try it in a virtual box first ;-)
Such scheme still does not support properly being truly cross platform, the needs of different systems/software (more fields, efficient filtering abilities), proper security methods (log items tied properly to the authentication/crypto stuff), more advanced centralized management features and so on...
Horrible. Plain horrible. Just like everything you can find at least from freshmeat.net and sf.net. Trust me, I have browsed through all the projects that contain the word "log" in their descriptions in any form. Absolutely none of them is worth anything.
In what way is an ascii pipe not cross platform?
Fields? What exactly do you mean by that?
Filtering? You can filter on regular expresions, hell, you can do bayesian filtering of events if you want.
Log items tied to auth/crypto? What log items should? please explain.
These mechanisms are much more flexible thatn you think.