Posts about qmail

Nothing Ever Really Goes Away On The Internet: ra-plugins

I used to manage a large number of QMail installations. And because Qmail was ... weirdly licensed, I wrote a set of plugins that ran on top of a patch called Qmail-SPP. I pretty much stopped doing that years ago because life took me in other directions, and forgot all about it.

That collection is called ra-plugins and I had not touched it since late 2008.

And today... I got a patch with two whole plugins to add to it so that it makes Qmail handle email addresses more like Gmail does (aliases using user+foo and making user.foo the same as userfoo).

So, I got them, added them, fixed a few simple building issues, updated the libsmtp it uses internally for one of the plugins to a later version, and there it stays, perhaps not to be touched until 2018.

Happy 10th blogiversary to me!

Since yesterday this blog is ten years old so, time for some history.

It all started in advogato where you could still read it today! (Please read it here instead ;-)

Then it moved to PyDS an early python desktop blog platform with a web interface, and was hosted in PyCS, a free service.

Then PyCS kinda died, and I started generating a static blog and hosting it in my ISP's free hosting. That sucked bad.

Then I started my own company, and I had my own servers, so I started hosting it there (even today this blog is completely static HTML!)

Then PyDS started acting weird, so I wrote my own blogging software, which is a real mess, perhaps 25% finished, but it does things exactly the way I like them.

Currently, this blog is syndicated in Planeta PyAr, Planet Python, Planet Qt, Planeta LUGLI, and a couple other places.

This year, I decided to make the blog completely bilingual (English and Spanish), but I hate translating it.

According to the stats I have available, the blog is in average more popular now than ever (but yes, my most popular posts were years ago ;-)

stats

These are the most popular pages in the last year:

Lessons:

  1. I need to write more about Qt and/or start flamewars with clueless IT writers
  2. I need to search for ancient material and deprecate it
  3. Having your own hosting and blogging software is neat
  4. 10 years is a lot of time: 860 posts (or 913, depending on how you count)

New qmail plugin idea: overload

It should not happen but it does: Your qmail server is overloaded. Maybe you are under a DOS attack, or there is a reason why you are getting 10x your usual amount of mail.

But then you start seeing how your "not preprocessed" queue starts growing, and growing...

This can also mean things like clamav or spamassassin, which need to check the mail before it gets queued are not keeping up with the mail flow, or maybe some IO performace issue.

But what can you do righ now to fix it?

Well, you can disable spamassassin, or, in extreme cases, shutdown SMTP so the system has a chance to catch its breath so to speak.

Of course, closing SMTP means your own users can't send email either, which sucks.

Now there is a lighter alternative: shutdown SMTP for those who are not your users.

Here's the trivial code, implemented as a SPP plugin, fit to be used in the [mail] section:

#!/bin/dash

if [ -f /var/qmail/control/overloaded ]
then
      if [ -z "$SMTPAUTHUSER" ]
      then
              echo R451 Temporary Failure: Server overload
              echo overload: $PPID Temporary Failure: Server overload >&2
      fi
fi

And if you are daring and want to make your system self-correcting, maybe you should cron something like this:

* * * * * if [ `qmail-qstat  | tail -1 | cut -d: -f2` -gt 100 ];\
then touch /var/qmail/control/overloaded ;\
else rm -f /var/qmail/control/overloaded; fi

I will probably code it again in C and make it part of ra/plugins.

If you are a qmail user: read this

  • If you don't know what qmail-spp is, please check it out. It makes qmail much much better.
  • If you know qmail-spp, then maybe my plugin collection will be handy for you.
  • My most useful plugin is probably ipthrottle, which you can use to make overeager IPs connect less often.
  • The version currently in SVN will autoblock those IPs for a configurable amount of time if you are using ipsvd which is like tcpserver, only much better.
  • I really need someone to help me test the SVN version, which should be way, way better than the releases on the page.
  • The SVN repo is at googlecode

Wanted: C programmer

Checking on my semi-dead projects, I found that one was almost finished but I had forgotten about it: rater

In order to make it really useful, however, I need a C programmer that can turn this python program:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
from socket import *
serverHost = 'localhost'
serverPort = 1999

s = socket(AF_INET, SOCK_STREAM)
s.connect((serverHost, serverPort))
print "Sending: ",' '.join(sys.argv[1:])
s.send(' '.join(sys.argv[1:])+"\n")
data = s.recv(1024)
sys.stderr.write(data)
sys.stderr.flush()
sys.exit(int(data.split(' ')[0]))

Into a nice function that never fails and never leaks memory (of course, it should return instead of exit, this is just example code ;-).

If that's done, I can release rater as a useful tool, which should find a home in many qmail installations (and maybe other uses).

Rater progresses (slowly)

I am hacking a bit on rater my daemon/client to see if things are happening more often than they should (in other words, generic rate limiting).

I had to take a few days off, since my brother got married and we all went back to Santa Fe for that and a weekend, and then everyone else has sore throats and I am the only one healthy.

But hey, it works well enough already:

  • The simplistic protocol is done
  • The server works
    • It can take hours of gibberish without problems.
    • It can take hours of valid input without problems.
    • It does what it's supposed to do.
  • It's staying below 300SLOC, which was my goal.

Missing stuff:

  • Valgrind it.
  • Client library.
  • Generic CLI client.
  • A qmail-spp plugin that uses it.

And then, I can forget all about it.

C is not Python II.

RaSPF, my C port of PySPF, is pretty much functional right now.

Here's what I mean:

  • It passes 75 internal unit tests (ok, 74 , but that one is arguable).
  • It passes 137 of 145 tests of the SPF official test suite.
  • It agrees with PySPF in 181 of the 183 cases of the libspf2 live DNS suite.
  • It segfaults in none of the 326 test cases.

So, while there are still some corner cases to debug, it's looking very good.

I even spent some time with valgrind to plug some leaks ( the internal test suite runs almost leakless, the real app is a sieve ;-)

All in all, if I can spend a little while with it during the week, I should be able to make a release that actually works.

Then, I can rewrite my SPF plugin for qmail, which was what sent me in this month-log tangent.

As a language wars comparison:

  • The sloccount of raspf is 2557 (or 2272 if we use the ragel grammar source instead of the generated file)
  • The sloccount of PySPF is 993.

So, a 2.6:1 or 2.28:1 code ratio.

However, I used 4 non-standard C libraries: bstrlib, udns, and helpers for hashes and exceptions, which add another 5794 LOCs.

So, it could be argued as a 8:1 ratio, too, but my C code is probably verbose in extreme, and many C lines are not really "logic" but declarations and such.

Also, I did not write PySPF, so his code may be more concise, but I tried my best to copy the flow as much as possible line-per-line.

In short, you need to write, according to this case, between 2 and 8 times more code than you do in Python.

That's a bit much!

Playing with literate programming

I am using ra-plugins as a toy to do things I never bothered in other projects.

I am doing unit-testing. And now... some literate programming!

Ok, not much, and not very well, but at least I am playing with Lp4all which is a nice, simple tool to generate nice HTML from slightly wiki-marked sources.

You can see some little things in my code here. My veredict so far? A nice way to keep the code documented in a fashion that ocasinal browsers can follow.

The main thing missing is automatic cross-referencing.

In general, I am finding that this (and unit testing) helps me express explicitly to myself what the heck I am trying to do, and see if the code actually does it. Which is a really good thing.

There is one thing worse than not having a test suite

UPDATE: There is *another* *better* test suite It is in YAML, though, so I need to parse it before I can use it, but that's my problem.

It's having a test suite that makes no sense.

I have written, for my ra-plugins project (you don't have to know what it is for this post anyway) a piece of code that tries to check mail senders using SPF.

SPF is an open standard. It has standard implementations. It has a test suite (http://www.schlitt.net/spf/tests/).

The test suite says this:
spfquery -ip=192.0.2.1 -sender=05.spf1-test.mailzone.com -helo=05.spf1-test.mailzone.com result /.*/ fail smtp-comment /.*/ explanation header-comment /.*/ spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender received-spf /.*/ Received-SPF: fail (spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender) client-ip=192.0.2.1; [email protected]; helo=05.spf1-test.mailzone.com;

So, yeah:

$ spfquery -ip=192.0.2.1 -sender=05.spf1-test.mailzone.com -helo=05.spf1-test.mailzone.com
fail
Please see http://www.openspf.org/why.html?sender=05.spf1-test.mailzone.com&ip=192.0.2.1&receiver=spfquery
spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender
Received-SPF: fail (spfquery: domain of 05.spf1-test.mailzone.com does not designate
192.0.2.1 as permitted sender) client-ip=192.0.2.1;
envelope-from=05.spf1-test.mailzone.com; helo=05.spf1-test.mailzone.com;

So, the standard implementation does what the test suite says.

Too bad that, if you bother checking the URL you are told to "please see"...

The domain 05.spf1-test.mailzone.com has published an SPF policy, however the policy is neutral on whether 192.0.2.1 is authorized to send mail on its behalf.

Either both the test suite and the sample implementation are wrong, or the site is wrong. And I am leaning towards "the test suite is wrong", because...

$ host -t txt 05.spf1-test.mailzone.com
05.spf1-test.mailzone.com descriptive text "v=spf1 default=deny"

If you check the record syntax (http://www.openspf.org/SPF_Record_Syntax) default is an unknown modifier, and should be ignored, so the record is simply "v=spf1", and indeed the result is neutral and there is no reason why this should be a fail.