A.P. Lawrence

Triple Threat

Good things come in small packages, but large things aren't always bad either. That must have been the thought someone had in designing an email newsletter for their customers. Apparently they had quite a bit of news; this particular piece of email was 34 megabytes when it arrived at the Mitel SME (E-Smith) server of one of my customers.

Although unreasonably large, that in itself wasn't a problem. While the original designers of qmail probably never expected anyone to send such a silly thing, the server accepted the mail. Unfortunately, for reasons unknown, the sending side of the transaction didn't seem to understand that it had been accepted. So every twenty minutes or so, it sent the mail again.

Well, that wasn't a terrible burden either. The server had plenty of disk space, and reasonable network bandwidth, so no harm yet. Not to the server anyway. The Windows machine of the actual recipient did start to have problems pretty quickly though: Microsoft Outlook Express soon literally started to choke as it clogged up with multiple 34 megabyte messages.

Forwarding rejected

There was another little glitch here that aggravated both the SME server and the poor Windows machine. The user also had her email forwarded by the SME to her home account. That was the intent, anyway. Unfortunately, her home isp wasn't about to accept a 34 megabyte mail message, so it rejected it back to the main account. That's how forwarding works: if your account on SME is fred@xyz.com and you forward to freddy@abc.com, if abc.com rejects the message, it goes back to fred@xyz.com. That caused yet another copy of this monster newsletter to come to her . Qmail would try to forward that again, but eventually would notice too many of its own headers and quit. Of course, the reject messages did make their way to Outlook.

Slowing Down

The toll eventually started to affect the SME server. The bandwidth was getting sucked away by the regular arrivals of 34 megabyte messages, which then were attempted to be forwarded to another server, which rejected them back, and more forwards were attempted, so more and more copies tried to go to Outlook, which was in no mood to swallow any more. It was a death spiral.

At this point, I called in for support from Mitel, and within a very few minutes had one of their engineers logged into the server.

The first necessity was to block the email. Fortunately, that's fairly easy to do. The SME server doesn't actually use qmail to recieve mail, but the smtpfront-qmail does recognize qmail control files. The Mitel engineer created /var/qmail/control/badmailfrom and added the sender's address to it:

newsletter@sillyplace.com

Be sure it's all lower case even if the sending mail says "Newsletter@SillyPlace.com".

Of course, smtpfront has to be stopped to put this in place. There are two ways to do that on SME:

svc -d /service/smtpfront-qmail
or
/etc/rc.d/init.d/smtpfront-qmail stop

If you examine the init.d script, you'll see that it actually does svc -d and svc -u.

The "svc" facility is used for both qmail and smtpfront-qmail on the 5.6 version of SME server. It's part of daemontools, a replacement for inittab (SME server still uses inittab; this is used in addition). To stop a daemontool service, use "svc -d"; to start it, use "svc -u".

This now blocked this person from sending mail, so the twenty minute cycles of new 34 megabyte newsletters was stopped.

What we now had was almost a dozen messages in the queue still trying to forward to the user's home ISP. To deal with those, we had to stop qmail (svc -d /service/qmail) and clean them out of the queue. That's not as straightforward with qmail as it might be, because qmail has a complicated queue structure.

Cleaning the queue

The actual program that uses the queue is "qmail-send". In the SME implementation, the svc -d /service/qmail kills that off. We are now free to mess with the queue. DON'T TOUCH THE QUEUE IF QMAIL IS RUNNING!!!

The actual messages will be found under /var/qmail/queue/mess in subdirectories. Each message will be stored with a name equal to its Queue ID. For example, you might have /var/qmail/queue/mess/21/1713843. If you examined it, you might see:

Received: (qmail 9013 invoked by uid 0); 17 May 2003 11:22:38 -0000
Date: 17 May 2003 11:22:38 -0000
Message-ID: <20030517112238.9012.qmail@aplawrence.org>
From: root@aplawrence.org
To: tony@aplawrence.org
Subject: ggg

Need eyes on the ground at your customer's site?
Installation and light training Boston and New England
Reliable and experienced, punctual and professional.

hhh

Other parts of the queue will have this same id:

# cd /var/qmail/queue
# find . -name 1713843
./queue/intd/1713843
./queue/mess/21/1713843
./queue/todo/1713843

So, to get rid of these newsletters, we simply need to find the queue id's and then remove them:

(The quotes around "cat /tmp/these" and "basename $i" are single backquotes - the unshifted "~" on most keyboards. The scripts WON'T WORK if you use regular single quotes!) AGAIN: DON'T TOUCH THE QUEUE IF QMAIL IS RUNNING!!!

cd /var/qmail/queue/mess
find . -exec grep -l "Newsletter" {} /dev/null \; > /tmp/these
cd /var/qmail/queue
for i in `cat /tmp/these`
do 
find . -name `basename $i` -exec rm {} \;
done

You could remove every ordinary file under the queue directory, but these need to stay under lock:

-rw-------    1 qmails   qmail           0 Oct 22 11:13 sendmutex
-rw-r--r--    1 qmailr   qmail        1024 Oct 22 11:15 tcpto
prw--w--w-    1 qmails   qmail          57 Oct 22 11:15 trigger

Restart qmail (svc -u /service/qmail) and smtpfront ( svc -u /service/smtpfront-qmail) it's done.

No more newsletter

Checking /var/log/smtpfront-qmail/current the next day showed that the constant attempts to send the newsletter stopped right after it was first rejected. To see dates in qmail log files, use "tai64nlocal":

tail /var/log/smtpfromt-qmail/current | tai64nlocal

It was then safe to remove the "badmailfrom" file and restart smtpfront-qmail a final time.

Enter your email address for automatic notification of new posts here
(be sure to whitelist 'feedburner.com' if you use spam filtering)

Control Spam and Viruses

Run your own mailserver

Have you tried Searching this site?

Unix/Linux/Mac OS X support by phone, email or on-site: Support Rates

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more. We appreciate comments and article submissions.

Publishing your articles here