2003-09-21 03:12

Queue Management for Qmail

Introduction

I am a big fan of qmail. I honestly think it's the best MTA around, even if it has a few issues. Well, what doesn't?

Since I manage a few small and large installations, I think it's only fair that I document what those issues seem to me, and what solutions I've found.

One thing about qmail is that its queue, for sake of efficiency, is not as easy to manipulate as, for example, sendmail's.

So, I've found some tools, and written some more, to help manage, control and manipulate qmail's message queue.

But first:

The Qmail Queue

When you use one of the tools described below, or look in qmail's logs, you will find things like mentions of "msg 457" being queued. What is that?

Well, the number is just an identifier, and you will find files called 457 in qmail's queue directories.

Qmail's queue consists of several directories. Some of them, which can be expected to hold many hundreds or thousands of files are split into numbered subdirectories. That is because in Unix, normally finding a file in a directory becomes slow if that directory contains many files.

A message going through qmail will have related files in several places in that directory structure.

The following is from qmail's INTERNALS file, some of these files will exist, some not:

mess/20/457
the message
todo/20/457
the envelope: where the message came from, where it's going
intd/20/457
the envelope, under construction by qmail-queue
info/20/457
the envelope sender address, after preprocessing
local/20/457
local envelope recipient addresses, after preprocessing
remote/20/457
remote envelope recipient addresses, after preprocessing
bounce/20/457
permanent delivery errors

The number of the internediate directory (20 in this example) is the remainder of dividing the message number by 23... at least by default ;-)

The numbers used to identify the messages are not arbirary and you should never change them unless you really know what you are doing. If you mess that up, maybe queue-rename (see below) can fix it.

If you remove a file, make sure you remove all the related ones, too, or qmail will whine.

If you gonna be messin' with qmail's queue, make sure you shut qmail-send down first. Bad things happen if you don't.

Tools that come with Qmail

qmail-qread
Lists the messages in the queue, it shows sender, recipient, date and message number.

[[email protected] root]# qmail-qread
16 Sep 2003 21:13:33 GMT  #145337  2218  <[email protected]>
        remote  [email protected]
qmail-qstat
Simple tool: Reports the number of messages in qmail's queue, and of messages waiting to enter the queue.

[[email protected] root]# qmail-qstat
messages in queue: 1025
messages in queue but not yet preprocessed: 2

Tools on the Net

QmHandle
Awesome tool. Combines the features of qmail-qstat and qmail-qread, adding functions to remove mails from the queue, display mails, dispatch queue immediately, treat local and remote queues separately, and more. A must have for any qmail admin.
queue-fix
A tool to automatically fix the problems explained above. You probably need to patch it to use big-todo (don't ask me)
queue-repair
Another tool with the same goal as queue-fix. Written in Python.
queue-rename
Renames the files in the queue structure so they have the proper names. If you move a qmail queue from one disk to another, you probably need to run this, or one of the queue-fixing tools.

Missing tool

qmail-cleaner
A tool to remove messages from the queue based on different criteria. qmHandle can do the hard work of actually removing, but which ones? Should be able to do it by sender, or by recipient, or by subject, or simply by regexp. Of course it is possible to do it using grep and shell, but that gets tricky when your queue exceeds a few hundred messages. Here's a first shot I made at writing such a tool.

Source code for qmail-cleaner.py

Important notes: This will simply print a list of the messages it finds matching your criteria. Since regular expressions are easy to get wrong, you should be careful about what you do with that list, because it could not be what you wanted....

In any case, it is easy to modify qmail-cleaner to output a qmHandle command line that would, for example, remove those messages from the queue. Such potentially destructive version is left as an exercise to the reader, though.

if you have any suggestions or problems with the script, please post a comment.

Source code for qmail-cleaner.py
                  
#!/usr/bin/env python

import sys,os,re

#Option handling nice and easy through Optik, http://optik.sf.net

from optik import OptionParser
parser = OptionParser()

parser.add_option ("-d", "--queue-dir",action="store", type="string", dest="queuedir",
                   help="The qmail queue directory, default=/var/qmail/queue",
                   default="/var/qmail/queue")

parser.add_option ("-s", "--sender",action="store", type="string", dest="sender",
                   help="Find messages with a From: header matching this regexp",
                   default=None)

parser.add_option ("-t", "--to",action="store", type="string", dest="to",
                   help="Find messages with a To: header matching this regexp",
                   default=None)

parser.add_option ("--header",action="store", type="string", dest="header",
                   help="Find messages with any header matching this regexp",
                   default=None)

parser.add_option("-v", action="store_true", dest="verbose", help="Verbose",default=0)
parser.add_option("-q", action="store_false", dest="verbose", help="Quiet")

(options,args) = parser.parse_args (sys.argv)


#This is where the messages are
msgdir=options.queuedir+"/mess"

#Array of matching messages
matches=[]

#Create the regexp to be matched
regex=""
if options.header: #User gave us a full regexp
        regex=options.header
elif options.sender: #User asks for a From: header
        regex="^From: "+options.sender
elif options.to: #User asks for a To: header
        regex="^To: "+options.to

exp=re.compile(regex)

if options.verbose:
        print "Searching messages with headers matching: %s\n"%regex

#Function that checks if a message matches our criteria
def check(message):
        global exp,matches
        for line in open(message):
                #Only check the headers, so stop on blank line
                if len(line)==1:
                        return

                #Check for match
                if exp.search(line,1):
                        matches.append(message.split("/")[-1])
                        return


#Get list of split directories (usually they are 23, but why bet)

try:
        dirs=os.listdir(msgdir)
except:
        sys.stdout.write("Error listing queue directories")
        sys.exit(1)


#Walk the tree and examine each file

for dir in dirs:
        files=os.listdir(msgdir+"/"+dir)
        for file in files:
                check (msgdir+"/"+dir+"/"+file)

#report files matching our criterion

for match in matches:
        print match

Sample run of qmail-cleaner.py:

[[email protected] p]# ./qmail-cleaner.py --header=ralsina  -v
Searching messages with headers matching: ralsina

145443
145307
146746
144469

Important Links

Comments

Comments powered by Disqus

Contents © 2000-2018 Roberto Alsina