When hosting companies accommodate increased traffic they’re likely to make changes to the network architecture and Dreamhost is no exception. Most recently, they separated the mail and hosting servers for better performance. On the upside my e-mail is much faster. On the downside my SpamAssassin (SA) installation is not working.

One solution for this problem is to use Dreamhost’s installation of SA. However, there’s no support for Bayesian rules. Those rules require a per-user database. Save whitelists, blacklists, and rule sensitivity, there’s no customization allowed. Another annoyance is logging into webmail to do anything meaningful with false positives; it’s explained further on the DH wiki.

Another option, and the best in my opinion, is using an IMAP client of my choice. This requires some modifications to my previous tutorial which takes you through the entire process of installing SA. Assuming that first tutorial is complete, you can now begin this tutorial. Yes, you will undo some of the work in the original tutorial. That’s the way it goes.

Because mail is being forwarded, moved, and synchronized all over the place, I thought a basic flowchart could simplify the end objective of this tutorial. As you can see, everything is setup to move e-mail to and from the mail server and hosting server.

A few assumptions before we get started…

• You have a basic understanding of Linux.
• You have an active DH account.
• You have access to the DH Panel.
• The %> notation indicates what you should type at the command prompt.
• The <Valid URL for: FileXYZ> notation indicates you need to insert a URL that points to the file in question.

1. Create a new fully hosted e-mail account. All of your e-mail aliases should be re-routed to this account’s address; this is now the gateway account through which everything must pass. You’ll have to create a similar account for each user with a SpamAssassin installation. The address doesn’t have to be user friendly, but choose something you can remember. In this example, I’ve created a new fully hosted address called spam_checker@unsaturated.com.
   
   If you have an alias called my.cool.address@unsaturated.com which forwards to your main user account, it will have to forward to spam_checker@unsaturated.com. Do not update all your aliases now because we want the transition to be seamless. Let’s keep e-mail flowing until all steps are complete.
   
2. Create a new forward-only e-mail address. After your mail is checked by SpamAssassin it will be forwarded to this new address, which then passes the e-mail along to your main user account, usually your_user_name@yourdomain.com. This step allows your e-mail to find its way back to the mail servers from the hosting servers. In this example, I’ve created a new forward-only address called spam_checker_passed@unsaturated.com.
   
3. Move the spam.rc file to your home folder. The spam.rc file was originally in a folder called procmail but it was the only file in there. This is a matter of preference but it’s important to keep this in mind for the rest of the tutorial. If nothing else is stored in your procmail folder, then delete it. Type the following at the command prompt:

   %> cd ~
   %> mv ~/procmail/spam.rc ~
   %> rm -Rf procmail/
   

4. Update the .procmailrc file in your home directory. You’ll notice several changes to this file, most notably all e-mail is forwarded to the address made in step 2 or then deleted. Remember, the spam has already been filtered to the .Spam folder according to the spam.rc file. Don’t forget to update the forwarding address! Type the following at the command prompt and insert the text shown:

   %> pico .procmailrc
   
   #=======================================
   # ~/.procmailrc
   #
   # Uses Maildir format mail directory.
   
   # Uncomment the following three lines to debug
   #LOGFILE=$HOME/procmail.log
   #VERBOSE=yes
   #LOGABSTRACT=all
   
   ## Directory for storing procmail-related files 
   PMDIR=$HOME 
   
   # Message directory (Courier IMAP and mutt)
   MAILDIR=$HOME/MaildirSync 
   
   # Spam filtering rules should run last
   INCLUDERC=$HOME/spam.rc 
   
   # Forward non-spam mail to validated address
   # REMEMBER TO UPDATE THIS ADDRESS!!!
   :0c
   ! spam_checker_passed@unsaturated.com
   
   # Everything should be filtered to the local .Spam folder 
   # or forwarded to the new mail server, so go ahead and 
   # delete whatever remains 
   :0
   /dev/null
   #=======================================
   

5. Rename the Maildir directory. Again, you might be wondering why this is necessary. In my opinion, the typical name for Maildir is okay if you’re actually using it for reading mail. However, this directory is now used exclusively for synchronizing mail accounts. Type the following at the command prompt:

   %> mv Maildir MaildirSync
   %> chmod -R 700 MaildirSync
   

6. Download and extract offlineimap. The features and speed of offlineimap looked compelling, so I tried it and decided to keep it. You can configure the script for many different scenarios but we’re keeping the steps basic. This step deletes the default configuration files but don’t worry because I’ll provided one later. Type the following at the command prompt:

   %> cd ~
   %> wget <Valid URL for: offlineimap_6.0.3.tar.gz>
   %> tar xvfz offlineimap_6.0.3.tar.gz
   %> cd offlineimap
   %> rm offlineimap.conf
   %> rm offlineimap.conf.minimal
   

7. Create an .offlineimaprc configuration file. Offlineimap needs to know some basic information like where to find your remote and local mail, folders to ignore, and more. To complete this you’ll need to know your mail server’s name, which can be accessed in the panel. You might wonder why this program is necessary. Consider a false positive e-mail, which is incorrectly marked spam. You could move that message to the correct folder but that change needs to be reflected on the hosting server where SpamAssassin can learn from the change. Remember to update all the values marked YOUR_. Type the following at the command prompt and insert the text shown:

   %> cd ~
   %> pico .offlineimaprc
   
   #=======================================
   # ~/.offlineimaprc
   
   [general]
   accounts = MainAccount
   metadata = ~/.offlineimap
   ignore-readonly = no
   ui = Noninteractive.Quiet
   
   [Account MainAccount]
   localrepository = Local
   remoterepository = Remote
   
   [Repository Local]
   type = Maildir
   localfolders = ~/MaildirSync
   
   [Repository Remote]
   type = IMAP
   remotehost = YOUR_MAIL_SERVER_NAME.mail.dreamhost.com
   ssl = true
   remoteuser = main_account@YOUR_DOMAIN.com
   remotepass = YOUR_PASSWORD
   nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
   folderfilter = lambda foldername: foldername in ['INBOX.Spam']
   maxconnections = 1
   holdconnectionopen = no
   #=======================================
   

8. Update the salearn.bat script. The old Bayes update script is obsolete because we have to ensure spam is synchronized between your hosting account and your mail account. This script synchronizes your .Spam folders, copies all messages (read and unread) to a single directory, tells SpamAssassin to update the Bayesian database, purges all the spam, then does a final synchronization. Type the following at the command prompt and insert the text shown:

   echo '========================================================'
   TESTED=false
   if [ "$1" ]
   then
     if [ $1 = "spam" ]
     then
       TESTED=true
       echo '--------------------------------------------------------'
       echo Synchronizing Spam folder...
   
       ~/offlineimap/offlineimap.py
       mv ~/MaildirSync/.Spam/new/* ~/MaildirSync/.Spam/cur/
   
       echo Messages synchronized and ready for processing.
       ~/sausr/bin/sa-learn -V
       echo Learning what is spam...
       ~/sausr/bin/sa-learn --spam ~/MaildirSync/.Spam/cur
       rm -f ~/MaildirSync/.Spam/cur/*
       echo Learning complete.  All spam messages were deleted.
       echo '--------------------------------------------------------'
       echo Resynchronizing your spam folder...
       ~/offlineimap/offlineimap.py
       echo All folders are synchronized. 
     elif [ $1 = "ham" ]
     then
       TESTED=true
       ~/sausr/bin/sa-learn -V
       echo '--------------------------------------------------------'
       echo Learning what is ham...
       ~/sausr/bin/sa-learn --ham ~/MaildirSync/cur  
     fi
     
     if [ $TESTED = true ]
     then
       echo '--------------------------------------------------------'
       echo Summary statistics of Bayes database...
       ~/sausr/bin/sa-learn --dump magic
       echo '--------------------------------------------------------'
     fi
   else
     echo Enter one argument:  [ham | spam]
   fi
   echo '========================================================'
   

9. Update your cron jobs. Because some e-mail are marked as spam incorrectly (false positives), we need to ensure all mail is synchronized. This gives you a chance to correct those false positives and move valid e-mail back into your inbox. The first cron task runs frequently to ensure your folders are up-to-date. Some spam systems would do this once daily but I prefer to know sooner if an important e-mail got trashed. The second task runs the Bayes update script. Type the following at the command prompt, acknowledge prompts for e-mail according to your preferences, and enter the text shown:

   %> crontab -e
   
   30 * * * * ~/offlineimap/offlineimap.py
   10 0 * * 7 ~/salearn.bat spam
   

10. Create an e-mail filter through the panel. Remember that e-mail account you created back in step 1 (spam_checker@unsaturated.com)? It’s time to give it a trivial filter using the panel.
    
    This will create a .procmailrc on the mail server, which you can’t see or edit via the shell. By trivial, I mean an obvious pass. Your rule would go something like this.
    

    Now once the trivial filter is establish you want to select the “Forward to shell account” option. Be sure to select the matching user account from the drop-down menu.
    

11. Update all e-mail aliases. Remember that e-mail account you created back in step 1 (spam_checker@unsaturated.com)? It’s time to update all your other aliases to point to that address. With each updated alias, the system goes “live” so you might try it on an infrequently used address, then send a test message.
    
12. Send a test message. If everything is working properly, you can run the salearn.bat script and you should be error-free. Send a test message and look for SA headers (X-Spam-Level, etc). Lastly, if you find an error in my tutorial please post a comment.

This entry was posted on Sunday, October 26th, 2008 at 12:19 PM and is archived under Projects. You can follow comments to this entry through the RSS 2.0 feed. You can leave a comment or trackback from your own site.