Junk (The MH-E Manual)

Marshall Rose once wrote a paper on MH entitled, How to process 200 messages a day and still get some real work done. This chapter could be entitled, How to process 1000 spams a day and still get some real work done.

We use the terms junk mail and spam interchangeably for any unwanted message which includes spam, viruses, and worms. The opposite of spam is ham. The act of classifying a sender as one who sends junk mail is called blocklisting; the opposite is called allowlisting.

The following table lists the options from the ‘mh-junk’ customization group.

The following option in the ‘mh-sequences’ customization group is also available.

MH-E depends on SpamAssassin, bogofilter, or SpamProbe to throw the dreck away. This chapter describes briefly how to configure these programs to work well with MH-E and how to use MH-E’s interface that provides continuing education for these programs.

The default setting of the option mh-junk-program is ‘Auto-detect’ which means that MH-E will automatically choose one of SpamAssassin, bogofilter, or SpamProbe in that order. If, for example, you have both SpamAssassin and bogofilter installed and you want to use bogofilter, then you can set this option to ‘Bogofilter’.

The command J b (mh-junk-blocklist) trains the spam program in use with the content of the range (see Ranges) and then handles the message(s) as specified by the option mh-junk-disposition. By default, this option is set to ‘Delete Spam’ but you can also specify the name of the folder which is useful for building a corpus of spam for training purposes.

In contrast, the command J a (mh-junk-allowlist) reclassifies a range of messages (see Ranges) as ham if it were incorrectly classified as spam. It then refiles the message into the +inbox folder.

If a message is in any sequence (except ‘Previous-Sequence:’ and ‘cur’) when it is allowlisted, then it will still be in those sequences in the destination folder. If this behavior is not desired, then turn off the option mh-allowlist-preserves-sequences-flag.

By default, the programs are run in the foreground, but this can be slow when junking large numbers of messages. If you have enough memory or don’t junk that many messages at the same time, you might try turning on the option mh-junk-background. ⁵³

The following sections discuss the various counter-spam measures that MH-E can work with.

SpamAssassin

SpamAssassin is one of the more popular spam filtering programs. Get it from your local distribution or from the SpamAssassin web site.

PATH=$PATH:/usr/bin/mh
MAILDIR=$HOME/`mhparam Path`

# Fight spam with SpamAssassin.
:0fw
| spamc

# Anything with a spam level of 10 or more is junked immediately.
:0:
* ^X-Spam-Level: ..........
/dev/null

:0:
* ^X-Spam-Status: Yes
spam/.

Note that one of the recipes above throws away messages with a score greater than or equal to 10. Here’s how you can determine a value that works best for you.

First, run ‘spamassassin -t’ on every mail message in your archive and use gnumeric to verify that the average plus the standard deviation of good mail is under 5, the SpamAssassin default for “spam”.

Using gnumeric, sort the messages by score and view the messages with the highest score. Determine the score which encompasses all of your interesting messages and add a couple of points to be conservative. Add that many dots to the ‘X-Spam-Level:’ header field above to send messages with that score down the drain.

In the example above, messages with a score of 5–9 are set aside in the ‘+spam’ folder for later review. The major weakness of rules-based filters is a plethora of false positives so it is worthwhile to check.

If SpamAssassin classifies a message incorrectly, or is unsure, you can use the MH-E commands J b (mh-junk-blocklist) and J a (mh-junk-allowlist).

The command J b (mh-junk-blocklist) adds a ‘blacklist_from’ entry to ~/spamassassin/user_prefs, deletes the message, and sends the message to the Razor, so that others might not see this spam. If the sa-learn command is available, the message is also recategorized as spam.

The commandJ a (mh-junk-allowlist) adds a ‘whitelist_from’ rule to ‘~/.spamassassin/user_prefs’. If the sa-learn command is available, the message is also recategorized as ham.

Over time, you’ll observe that the same host or domain occurs repeatedly in the ‘blacklist_from’ entries, so you might think that you could avoid future spam by blocklisting all mail from a particular domain. The utility function mh-spamassassin-identify-spammers helps you do precisely that. This function displays a frequency count of the hosts and domains in the ‘blacklist_from’ entries from the last blank line in ~/.spamassassin/user_prefs to the end of the file. This information can be used so that you can replace multiple ‘blacklist_from’ entries with a single wildcard entry such as:

blacklist_from *@*amazingoffersdirect2u.com

In versions of SpamAssassin (2.50 and on) that support a Bayesian classifier, J b (mh-junk-blocklist) uses the program sa-learn to recategorize the message as spam. Neither MH-E, nor SpamAssassin, rebuilds the database after adding words, so you will need to run ‘sa-learn --rebuild’ periodically. This can be done by adding the following to your crontab:

0 * * * *       sa-learn --rebuild > /dev/null 2>&1

Bogofilter

Bogofilter is a Bayesian spam filtering program. Get it from your local distribution or from the bogofilter web site.

bogofilter -n < good-message

bogofilter -s < spam-message

on every spam message. This is called a full training; three other training methods are described in the FAQ that is distributed with bogofilter. Note that most Bayesian filters need 1000 to 5000 of each type of message to start doing a good job.

PATH=$PATH:/usr/bin/mh
MAILDIR=$HOME/`mhparam Path`

# Fight spam with Bogofilter.
:0fw
| bogofilter -3 -e -p

:0:
* ^X-Bogosity: Yes, tests=bogofilter
spam/.

:0:
* ^X-Bogosity: Unsure, tests=bogofilter
spam/unsure/.

If bogofilter classifies a message incorrectly, or is unsure, you can use the MH-E commands J b (mh-junk-blocklist) and J a (mh-junk-allowlist) to update bogofilter’s training.

The Bogofilter FAQ suggests that you run the following occasionally to shrink the database:

bogoutil -d wordlist.db | bogoutil -l wordlist.db.new
mv wordlist.db wordlist.db.prv
mv wordlist.db.new wordlist.db

SpamProbe

SpamProbe is a Bayesian spam filtering program. Get it from your local distribution or from the SpamProbe web site.

PATH=$PATH:/usr/bin/mh
MAILDIR=$HOME/`mhparam Path`

# Fight spam with SpamProbe.
:0
SCORE=| spamprobe receive

:0 wf
| formail -I "X-SpamProbe: $SCORE"

:0:
*^X-SpamProbe: SPAM
spam/.

If SpamProbe classifies a message incorrectly, you can use the MH-E commands J b (mh-junk-blocklist) and J a (mh-junk-allowlist) to update SpamProbe’s training.

Other Things You Can Do

There are a couple of things that you can add to ~/.procmailrc in order to filter out a lot of spam and viruses. The first is to eliminate any message with a Windows executable (which is most likely a virus). The second is to eliminate mail in character sets that you can’t read.

PATH=$PATH:/usr/bin/mh
MAILDIR=$HOME/`mhparam Path`

#
# Filter messages with w32 executables/virii.
#
# These attachments are base64 and have a TVqQAAMAAAAEAAAA//8AALg
# pattern. The string "this program cannot be run in MS-DOS mode"
# encoded in base64 is 4fug4AtAnNIbg and helps to avoid false
# positives (Roland Smith via Pete from the bogofilter mailing list).
#
:0 B:
* ^Content-Transfer-Encoding:.*base64
* ^TVqQAAMAAAAEAAAA//8AALg
* 4fug4AtAnNIbg
spam/exe/.

#
# Filter mail in unreadable character sets (from the Bogofilter FAQ).
#
UNREADABLE='[^?"]*big5|iso-2022-jp|ISO-2022-KR|euc-kr|gb2312|ks_c_5601-1987'

:0:
* 1^0 $ ^Subject:.*=\?($UNREADABLE)
* 1^0 $ ^Content-Type:.*charset="?($UNREADABLE)
spam/unreadable/.

:0:
* ^Content-Type:.*multipart
* B ?? $ ^Content-Type:.*^?.*charset="?($UNREADABLE)
spam/unreadable/.

Footnotes

(53)

Note that the option mh-junk-background is used as the destination argument in the call to call-process. Therefore, turning on this option means setting its value to ‘0’. You can also set its value to ‘t’ to direct the programs’ output to the *MH-E Log* buffer; this may be useful for debugging.

19 Dealing With Junk Mail