It should not happen but it does: Your qmail server is overloaded. Maybe you are under a DOS attack, or there is a reason why you are getting 10x your usual amount of mail.
But then you start seeing how your "not preprocessed" queue starts growing, and growing...
This can also mean things like clamav or spamassassin, which need to check the mail before it gets queued are not keeping up with the mail flow, or maybe some IO performace issue.
But what can you do righ now to fix it?
Well, you can disable spamassassin, or, in extreme cases, shutdown SMTP so the system has a chance to catch its breath so to speak.
Of course, closing SMTP means your own users can't send email either, which sucks.
Now there is a lighter alternative: shutdown SMTP for those who are not your users.
Here's the trivial code, implemented as a SPP plugin, fit to be used in the [mail] section:
#!/bin/dash
if [ -f /var/qmail/control/overloaded ]
then
if [ -z "$SMTPAUTHUSER" ]
then
echo R451 Temporary Failure: Server overload
echo overload: $PPID Temporary Failure: Server overload >&2
fi
fi
And if you are daring and want to make your system self-correcting, maybe you should cron something like this:
* * * * * if [ `qmail-qstat | tail -1 | cut -d: -f2` -gt 100 ];\
then touch /var/qmail/control/overloaded ;\
else rm -f /var/qmail/control/overloaded; fi
I will probably code it again in C and make it part of ra/plugins.