Posts about ra-plugins

New qmail plugin idea: overload

It should not happen but it does: Your qmail server is overloaded. Maybe you are under a DOS attack, or there is a reason why you are getting 10x your usual amount of mail.

But then you start seeing how your "not preprocessed" queue starts growing, and growing...

This can also mean things like clamav or spamassassin, which need to check the mail before it gets queued are not keeping up with the mail flow, or maybe some IO performace issue.

But what can you do righ now to fix it?

Well, you can disable spamassassin, or, in extreme cases, shutdown SMTP so the system has a chance to catch its breath so to speak.

Of course, closing SMTP means your own users can't send email either, which sucks.

Now there is a lighter alternative: shutdown SMTP for those who are not your users.

Here's the trivial code, implemented as a SPP plugin, fit to be used in the [mail] section:

#!/bin/dash

if [ -f /var/qmail/control/overloaded ]
then
      if [ -z "$SMTPAUTHUSER" ]
      then
              echo R451 Temporary Failure: Server overload
              echo overload: $PPID Temporary Failure: Server overload >&2
      fi
fi

And if you are daring and want to make your system self-correcting, maybe you should cron something like this:

* * * * * if [ `qmail-qstat  | tail -1 | cut -d: -f2` -gt 100 ];\
then touch /var/qmail/control/overloaded ;\
else rm -f /var/qmail/control/overloaded; fi

I will probably code it again in C and make it part of ra/plugins.

If you are a qmail user: read this

  • If you don't know what qmail-spp is, please check it out. It makes qmail much much better.
  • If you know qmail-spp, then maybe my plugin collection will be handy for you.
  • My most useful plugin is probably ipthrottle, which you can use to make overeager IPs connect less often.
  • The version currently in SVN will autoblock those IPs for a configurable amount of time if you are using ipsvd which is like tcpserver, only much better.
  • I really need someone to help me test the SVN version, which should be way, way better than the releases on the page.
  • The SVN repo is at googlecode

Rater progresses (slowly)

I am hacking a bit on rater my daemon/client to see if things are happening more often than they should (in other words, generic rate limiting).

I had to take a few days off, since my brother got married and we all went back to Santa Fe for that and a weekend, and then everyone else has sore throats and I am the only one healthy.

But hey, it works well enough already:

  • The simplistic protocol is done
  • The server works
    • It can take hours of gibberish without problems.
    • It can take hours of valid input without problems.
    • It does what it's supposed to do.
  • It's staying below 300SLOC, which was my goal.

Missing stuff:

  • Valgrind it.
  • Client library.
  • Generic CLI client.
  • A qmail-spp plugin that uses it.

And then, I can forget all about it.

Snow and rates

Monday was a very special day:

  • Holiday (Independence day)
  • Anniversary (3 years as Rosario's boyfriend)
  • The first snowfall in Buenos Aires in 89 years.

Besides that, this week my brother is getting married so the whole family (including 2.5 month-old JF) is leaving for my ancestral lands tomorrow.

And I started a new small project, whcih should be finished soon.

This is something that seems useful to me in the context of mail servers, but maybe it will also find its uses elsewhere.

I call it rater, and it tells you if things are happening faster than a specific rate.

For example, I intend to use it to figure out if a specific IP is connecting to a server more than X times every Y seconds, or if a user is sending more than Z emails every T minutes.

The only thing I found for this is relayd, which is old, unmaintained and whose site has vanished.

The config file is something like this (thanks to libconfig):

limits : {
      user: (
                    ("rosario",90,20),
                    ("ralsina",90,10),
              ("*",2,10)
              );
      ip:   (
                    ("10.0.0.*",90 , 20),
                    ("10.0.1.*",90 , 20),
                    ("*",2 , 10)
              );

};

You can define as many classes of limits as you want (that would be ip and user in this example) and as many limit keys as you want, that will be matched using something like fnmatch.

I am using an in-memory SQLite DB for the accounting, and an interesting library called libut for the sockets, logging, and event loop.

This library has a very interesting feature: your app gets an administrative interface for free!

[[email protected] rater]$ telnet localhost 4445
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
libut control port interpreter
Type 'help' for command list.

help
command           description
----------------- -------------------------------------
* mem             - memory pool usage summary
* var             - Display or set config variables
* log             - Change log file or verbosity
  fds             - list selected file descriptors
  tmr             - show pending timers
  uptime          - show uptime
* prf             - Performance/profiling stats
* cops            - List coprocesses
  help            - view command help
  exit            - close connection

Commands preceded by * have detailed help. Use help <command>.

Ok
var
 name                 description                    value
--------------------- ------------------------------ --------------------
*ut_log_level         log level                      Debugk
*ut_log_file          log file                       /dev/stdout
*ut_jobname           job name                       job1
*ut_control_port      control port IP/port           127.0.0.1:4445
*ut_basedir           shl base directory             /mnt/centos/home/ralsina/Desktop/proyectos/rater

Variables prefixed with '*' can be changed.

Ok
var ut_log_level Debug

Ok
var
 name                 description                    value
--------------------- ------------------------------ --------------------
*ut_log_level         log level                      Debug
*ut_log_file          log file                       /dev/stdout
*ut_jobname           job name                       job1
*ut_control_port      control port IP/port           127.0.0.1:4445
*ut_basedir           shl base directory             /mnt/centos/home/ralsina/Desktop/proyectos/rater

Variables prefixed with '*' can be changed.

Ok

Pretty neat.

Beyond this, there will be a small client-side library that hides all the network stuff behind a couple of blocking calls (or you can do your own because the protocol is silly simple).

Playing with literate programming

I am using ra-plugins as a toy to do things I never bothered in other projects.

I am doing unit-testing. And now... some literate programming!

Ok, not much, and not very well, but at least I am playing with Lp4all which is a nice, simple tool to generate nice HTML from slightly wiki-marked sources.

You can see some little things in my code here. My veredict so far? A nice way to keep the code documented in a fashion that ocasinal browsers can follow.

The main thing missing is automatic cross-referencing.

In general, I am finding that this (and unit testing) helps me express explicitly to myself what the heck I am trying to do, and see if the code actually does it. Which is a really good thing.

There is one thing worse than not having a test suite

UPDATE: There is *another* *better* test suite It is in YAML, though, so I need to parse it before I can use it, but that's my problem.

It's having a test suite that makes no sense.

I have written, for my ra-plugins project (you don't have to know what it is for this post anyway) a piece of code that tries to check mail senders using SPF.

SPF is an open standard. It has standard implementations. It has a test suite (http://www.schlitt.net/spf/tests/).

The test suite says this:
spfquery -ip=192.0.2.1 -sender=05.spf1-test.mailzone.com -helo=05.spf1-test.mailzone.com result /.*/ fail smtp-comment /.*/ explanation header-comment /.*/ spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender received-spf /.*/ Received-SPF: fail (spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender) client-ip=192.0.2.1; [email protected]; helo=05.spf1-test.mailzone.com;

So, yeah:

$ spfquery -ip=192.0.2.1 -sender=05.spf1-test.mailzone.com -helo=05.spf1-test.mailzone.com
fail
Please see http://www.openspf.org/why.html?sender=05.spf1-test.mailzone.com&ip=192.0.2.1&receiver=spfquery
spfquery: domain of 05.spf1-test.mailzone.com does not designate 192.0.2.1 as permitted sender
Received-SPF: fail (spfquery: domain of 05.spf1-test.mailzone.com does not designate
192.0.2.1 as permitted sender) client-ip=192.0.2.1;
envelope-from=05.spf1-test.mailzone.com; helo=05.spf1-test.mailzone.com;

So, the standard implementation does what the test suite says.

Too bad that, if you bother checking the URL you are told to "please see"...

The domain 05.spf1-test.mailzone.com has published an SPF policy, however the policy is neutral on whether 192.0.2.1 is authorized to send mail on its behalf.

Either both the test suite and the sample implementation are wrong, or the site is wrong. And I am leaning towards "the test suite is wrong", because...

$ host -t txt 05.spf1-test.mailzone.com
05.spf1-test.mailzone.com descriptive text "v=spf1 default=deny"

If you check the record syntax (http://www.openspf.org/SPF_Record_Syntax) default is an unknown modifier, and should be ignored, so the record is simply "v=spf1", and indeed the result is neutral and there is no reason why this should be a fail.

ra-plugins 0.2.9 is coming closer

Version 0.2.9 of ra-plugins, my qmail-spp plugin collection is coming soon.

Including lots and lots of new plugins, a real build system, and even two patches by someone else :-)

So, now is a good time to let me know if you are using ra-plugins, if you have any problems with it, and if you have any ideas for cool plugins. I can write them.

Ideas for SMTP plugins

The only current software I wrote that some people actually use is called RA-Plugins. It's a series of proggies you plug in your SMTP server's conversation, and do diverse things with it, like rejecting messages that fail certain criteria, checking the status of the recipient's account, whatever.

Injecting this stuff in the middle of SMTP is good because it means you will reject the messages before they get into your server. But... I am running out of ideas, so... have any you can spare? :-)

You can see the current plugin list here and the only ideas I have left are:

  • A plugin that calls back to the sender's email server and tries to email him, ala milter-sender to catch forged senders.
  • A plugin to autowhitelist in spamassassin those addresses to which you send mail.
  • A plugin to keep an account of how many connections you hold to each IP, and limit them. (Not currently possible)

If you understand what I wrote, and have any ideas... feel free to post them as comments and/or email me with them!

CMake is nice. Or not?

I am playing with CMake. Specifically, I am trying to replace the simplistic handmade Makefiles for my RA-Plugins project.

The parts about detecting libraries and conditionally compiling plugins based on what you have were surprisingly easy!

Until I ran into ... man pages.

Here is a Makefile that would build all the manpages for the plugins:

MANPAGES=plugins-ra.8 authchecks.8 rcptchecks.8
man: \$(MANPAGES)
%.8: %.man.txt
      txt2man -t \${basename $< .man.txt} < $< > [email protected]

As you can see... trivial. All I need to do in order to add a man page is add whatever.8 to the list, and it will be created from whatever.man.txt.

But... how does one do that using CMake?

I started thinking of using a FILE (GLOB *.man.txt) and then a FOREACH over that, and then... then what? Really, I am stumped. I am a newbie, though, and getting the big, difficult stuff done is enough to switch. I should generate these before distribution anyway.

So, I wrote a wee Makefile.manpages and added this for CMAKE:

ADD_CUSTOM_TARGET ( manpages make -f Makefile.manpages )

But I am curious about finding the cmakeish way.