Making Linux systems that don't suck. Part I

Posted: 2007-07-21 21:28 | Leer en español | More posts about sysadmin linux

I wrote in a previous post that there is a whole lot of software shipped with most linux distros which sucks (although I said a it a bit more mildly).

You could consider an old post of mine, called Logging in style as the 0th part of this series. Please read it first. It's probably better than this one.

Now, I will tackle another, related area... How does your system tell you what happened?

You may say (specially if you read the piece I mentioned) "Through the system logs!". And you are right. Except that doing it that way is crazy.

Mind you, logs are not crazy, it's relying on them to know what happens that's crazy.

Logs are great to know what happened when you know what you are looking for. Not a way to know what happened that was important.

Right now, the tool your Linux box uses to tell you what happened is probably sendmail [1].

Here are a few examples:

Every programmed task using at or cron mails its output to the administrator. If you do any sysadmin work you know this is very important.
Logwatch is run by cron, examines your logs and gives you a report on system activity. And mails it.
Bacula (my favourite serious backup tool) runs its backup and restore jobs, then mails the output to the backup administrators.
Munin or Argus monitor systems and services on many computers, and mail you if things go wrong.

And so on, and so forth. Almost every tool that needs to alert you if something happened, or needs to report to you the results of a scheduled task, will end up mailing you.

This is specially annoying in that the default configuration in all Linux Distros is to do this kind of things in a way that can't possibly be what you want: Mailing this stuff to the local root account.

Let's consider some of the ways in which this is wrong...

Usually you can't read root's mail using POP or IMAP (and you really shouldn't) [2].
So you need to read it locally. Using what? mail(1)? mutt? Or are you asking the user to login at the GUI as root? At least forward root's mail to the user account you create on installation, please! (no idea if any distro does this).
If you never bother reading and deleting those mails, they will eventually fill your / or /home partition (if you use ~/Maildir as mail store) or your /var partition (for /var/spool/mail store).

Suppose you know this already. You fix it. [3] You forward root's mail to an account you actually read. Usually that should mean a remote account, of course.

Ignoring the fact that mail delivery is getting less and less reliable every day (at least don't use a hotmail account, ok?) let's assume you are getting all this administrative mail.

What's the problem now?

That you are running a general purpose mail server. On every box. All the time. Probably allowing mail for every account in the system, and usually opening a port, even if only in the loopback interface.

Does that make any sense? Why do you have a service so complex and annoying installed? Why use a software you can buy books about? Is it necessary? I say no. We'll see if I'm right.

Now, back to the reliability of mail issue: You need to configure a mail server right, or use a smart host as a mail gateway, or you are going to lose mail.

If you are going to ignore everything else in this post, don't ignore this: use a smart host. Please!

We have enough malformed mail floating around. We don't need another million messages from root@localhost to [email protected].

If you can, use your own domain's mail server as smarthost for your boxes, so noone needs to forward your mail.

If you can't do that, hire a virtual box from tektonic, set up a receive-only mail server with authenticated SMTPS and/or access limited to specific fixed IP addresses, and use that as a smart host so your administrative mail doesn't float around causing troubles for us onlookers.

Now let's get constructive. What can we do about it?

The first good thing is that there is really no reason why all these notifications need to be sent via email.

What's happening is that all these different applications are probably using sendmail. But that's just sendmail the command, not sendmail the MTA or any other MTA. The use of the MTA is just a side effect.

So, we have a simple path ahead of us:

Decide how you want to receive your notifications. Store tham in a DB? Send you IM messages? Read them aloud? Store files in a folder structure? Show popup dialogs?
Implement a sendmail interface that does exactly that.

I should probably add: does exactly that in the simplest way possible.

I have not yet thought through step 1, though. So, I am still using email. But for that specific case, I have step 2 figured out: nullmailer.

Usually you need to edit only two lines to make it work nicely with a smarthost (a working configuration is probably about 4 lines).
Its size is a whooping 227KB in my box.
The detailed docs are about 300 terminal lines long (The HOWTO is 54 lines).
It uses almost no resources whatsoever when not sending mail: about 1MB of ram (450KB RSS) and 1 CPU-second a day on my notebook.
It will probably force you to use runit or daemontools and a decent logger even if it's only for this one service ;-)

Why are no distros shipping this or something like it? Do this, add a trivial configuration tool, and your computer is suddenly 10 times better cared for.

If I were managing a commercial distro, I would give mail accounts to all my paying customers and ship all boxes with a default setting of "put the reports on this webpage" (to be enabled by the user). It would cost almost nothing, and prevent part of the "I never knew 25 people logged via SSH on my root account yesterday" syndrome.

And I would really think hard about what a good delivery mechanism is for this kind of messages, and implement it. Because right now... it sucks. Come on, IBM/RH/Ubuntu/Whoever, this is not hard stuff, this is low hanging fruit. Show some gumption and pick it up!

Join me next week when I will explain why cron and at suck.

[1]	This is true at least on RHEL4 and several other "modern" Linuxes. But even if it's Postfix or Exim: it's wrong anyway.

[2]	And if you misconfigure your POP/IMAP server so you can do it, you just hacked your own box.

[3]	It would be better if you didn't have to unbreak the system, but let's ignore that for a while.