Using Amazon’s SES and Sendmail for mailing lists

I have run a private mailing list for years, and recently migrated to Amazon’s SES service. While some documentation from Amazon is available, it is far from complete. Here are some of the changes and improvements I made.

First, you will need to do the standard steps for setting up the SES service. I used Amazon’s AWI image as a starting point and I followed this guide.

But I hit a few glitches. Here’s what I did to fix them.

So first – make sure you have the necessary software.

sudo yum update
sudo yum upgrade
sudo yum install emacs  # Just a personal preference
sudo yum install mailx sendmail-cf m4 
sudo yum install sendmail-milter dkim-milter spamassassin

I asked for an EC2 Elastic IP and associated my instance of Amazon Linux AWI to use this IP address.

I validated a test email address as described here.

I set up my /etc/mail/authinfo file

I set up my sendmail to forward to Amazon by adding this to my sendmail.mc file:

define(`SMART_HOST', `email-smtp.us-east-1.amazonaws.com')dnl

Amazon’s SES SMARTHOST : Maximum sending rate exceeded

One of my early tests of the mailing list was frustrating. I have several hundred people on one list, and when I tried to send mail, the first 50 emails were accepted, and the rest were rejected because Amazon has a maximum of 50 emails in any single message. The actual error I saw was:

   ----- Transcript of session follows -----
 ... while talking to email-smtp.us-east-1.amazonaws.com:
 >>> DATA
 <<< 454 Throttling failure: Maximum sending rate exceeded.

In addition, the outgoing mail queue was only being emptied once an hour. For a system sending email, this was bad news. But here’s the solution:

I modified my /etc/sysconfig/sendmail

#!/bin/bash
DAEMON=yes                                                                                    
QUEUE=10m

This runs a daemon to send outgoing mail, and runs it once every 10 minutes Iinstead of once an hour). And I added the following to my sendmail.mc file

define(`SMTP_MAILER_MAXMSG',`20')
define(`SMTP_MAILER_MAXRCPTS',`40')
define(`RELAY_MAILER_MAXMSG',`20')
define(`RELAY_MAILER_MAXRCPTS',`40')

These are documented on the sendmail site. The above will specify that the SMTP and RELAY mailer have 40 maximum recipients, and to deliver 20 messages maximum. Perhaps I could increase the number 20, but I tend to be conservative in my selections.

Modifying /etc/mail/access

You will need to add entries to the /etc/mail/access file to specify the connections

Connect:localhost.localdomain           RELAY
Connect:localhost                       RELAY
Connect:127.0.0.1                       RELAY
Connect:email-smtp.us-east-1.amazonaws.com      RELAY

Modifying/etc/mail/local-host-names

You must add all aliases for your mail server to the above file. Otherwise, you will get a “relaying denied” error.

Using mailertable

The mailertable option is very useful when debugging your mail forwarding. Normally, sendmail treats all email the same – either relay it to a SMARTHOST, or directly send it using the ESMTP mailer. Having the option to modify this based on the domain gives you flexibility. To do this, you need to add the following file to your sendmail.mc file

FEATURE(`mailertable', `hash -o /etc/mail/mailertable')dnl

I have a gmail address I was using for testing, so I added the following lines to this file

gmail.com       esmtp:gmail.com

For each domain, either use “esmtp” or “relay”. For gmail, I used the “esmtp” mailer.

I added the file /etc/mail/MakeMailertable that contains

#!/bin/sh
sudo makemap hash /etc/mail/mailertable.db < /etc/mail/mailertable

This makes it easier to rebuilt just the database, instead of restarting sendmail. I did the same for the authinfo file.

That’s all you need to do to add the mailertable option. However you have to make sure the file permissions are correct, along with everything else.  This is how you debug the mailertable function:

 echo "3,0 user@gmail.com" | /usr/lib/sendmail -bt -d60.1 -d38.2

The expected output, when all works well, is:

seq_map_parse(aliases.files, )
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> canonify           input: user @ gmail . com
Canonify2          input: user < @ gmail . com >
map_lookup(host, gmail.com) => gmail.com. (0)
Canonify2        returns: user < @ gmail . com . >
canonify         returns: user < @ gmail . com . >
parse              input: user < @ gmail . com . >
Parse0             input: user < @ gmail . com . >
Parse0           returns: user < @ gmail . com . >
Parse1             input: user < @ gmail . com . >
hash_map_open(mailertable, /etc/mail/mailertable, 0)
map_lookup(mailertable, gmail.com) => esmtp:gmail.com (0)
MailerToTriple     input: < esmtp : gmail . com > user < @ gmail . com . >
MailerToTriple   returns: $# esmtp $@ gmail . com $: user < @ gmail . com . >
Parse1           returns: $# esmtp $@ gmail . com $: user < @ gmail . com . >
parse            returns: $# esmtp $@ gmail . com $: user < @ gmail . com . >

Installing milter-regex

One of the best ways I have found to reduce the amount of incoming spam is to add the regular expression milter filter in sendmail.

In particular, I have configured by server to examine the incoming SMTP connection, where the sending system announces its hostname using the HELO message.  I add a test that verifies that the hostname is a FQDN (fully qualified domain name), and that the reverse lookup matches the IP address of the host name. In short, compromised systems sending spam rarely even provide the proper hostname of the system when they generate spam, so my system never accepts the message in the first place. Technically, this violates the RFCs, but I have done this for more than 10 years, and when combined with the various spam blocklists, the milter-regex solution eliminates most of the spam from spambots.

One way to do this is a new feature in sendmail 8.14 in the sendmail.mc file:

FEATURE(`require_rdns')dnl

As I’ve been using milter-regex for a decade, I documented this method below. I like this as I can add special rules very easily.

However, the installation is a little tricky. You need to install the source of sendmail, as you have to recompile milter-regex. Before you do this, you need some compilation tools:

sudo yum install gcc byacc # needed GCC and bison to build libmilter

Now get the source of sendmail and set up some directories. I generally capitalize my directories, to distinguish them from standard directories

cd ~/Src/Sendmail
wget ftp://ftp.sendmail.org/pub/sendmail/PGPKEYS
gpg --import PGPKEYS
wget ftp://ftp.sendmail.org/pub/sendmail/sendmail-current.tar.gz
wget ftp://ftp.sendmail.org/pub/sendmail/sendmail-current.tar.gz.sig
gpg --verify sendmail-current.tar.gz.sig  sendmail-current.tar.gz
# Now let's unpack the sources
tar xvf sendmail-*.gz
ln -s ./sendmail.8.14.7 ./sendmail # I'll use this link later
wget http://www.benzedrine.cx/milter-regex-1.9.tar.gz
md5sum milter-regex-1.9.tar.gz
tar xvf milter-regex-1.9.tar.gz

You should  verify the milter-regex signature using the milter-regex site

md5sum milter-regex-1.9.tar.gz                                     |

Which returns

65471e363669b237e280d53dc914ac2a  milter-regex-1.9.tar.gz

I also set up some symbolic links. Note that I link to ../sendmail which is a link to the current version of sendmail. If I update sendmail’s sources, I just need to change one symbolic link (i.e. sendmail -> sendmail.current), and then I can recompile the milter-regex source without changing the links.

cd milter-regex-1.9
ln -s ../sendmail/libmilter/
ln -s ../sendmail/include/
ln -s ../sendmail/sm/
ln -s Makefile.linux Makefile # for convenience

I found a few compile errors.  I modified Makefile.linux so that some lines were changed from

CFLAGS=               -g
LDFLAGS=      -L/usr/lib/libmilter -lmilter -lpthread

to

CFLAGS=               -g -I../sendmail/
LDFLAGS=       -lmilter -lpthread
YACC=bison -y

and

yacc -d parse.y

to

${YACC} -d parse.y

I also had to make sure I could find libmilter, which wasn’t in the expected location

cd /usr/lib                                                 
sudo ln -s ../lib64/libmilter.so.1.0 ./libmilter.so

Now the command “make” generates

gcc -g -I../sendmail/include  -c milter-regex.c
gcc -g -I../sendmail/include  -c eval.c
gcc -g -I../sendmail/include  -c strlcat.c
gcc -g -I../sendmail/include  -c strlcpy.c
bison -y -d parse.y
gcc -g -I../sendmail/include  -c y.tab.c
gcc -o milter-regex milter-regex.o eval.o strlcat.o strlcpy.o y.tab.o -lmilter -lpthread
nroff -Tascii -mandoc milter-regex.8 > milter-regex.cat8

Good. Next, I “installed it” by

sudo cp milter-regex /usr/bin
sudo cp -i milter-regex.8  /usr/share/man/man8

I also needed to have a process that is running at bootup time. But I run it as the user “milter-regex” created by the useradd command.

cd /etc/rc3.d
sudo ln -s ../init.d/milter-regex S70milter-regex

And here is my /etc/init.d/milter-regex file:

#!/bin/sh
# $Id: milter-regex.init,v 1.1.1.1 2007/01/11 15:49:52 dhartmei Exp $
#  init file for milter-regex
#  modified by Jim Klimov from init file for milter-greylist, (C) Mar 2005
#
# chkconfig: - 50 50
# description: Milter Regex Daemon
#
# processname: /usr/bin/milter-regex
# config: /etc/mail/milter-regex.conf
# pidfile: /var/milter-regex/milter-regex.pid

# source function library
. /etc/init.d/functions

configfile="/etc/mail/milter-regex.conf"
basedir="/var/spool/milter-regex"
# pidfile="$basedir/milter-regex.pid"
#socket="$basedir/milter-regex.sock"
socket="$basedir/sock"
user="milter-regex"
OPTIONS="-c $configfile -u $user -p $socket"

# Enable testing only for users specified in config file
# OPTIONS="-T $OPTIONS"

# No idea what we could put here... paths like above? ;)
if [ -f /etc/sysconfig/milter-regex ]
then
    . /etc/sysconfig/milter-regex
fi
RETVAL=0
prog="Milter-Regex"

start() {
        echo -n $"Starting $prog: "
        if [ $UID -ne 0 ]; then
                RETVAL=1
                failure
        else
                if [ x"$basedir" != x -a x"$basedir" != x/ ]; then
                    if [ ! -d "$basedir" ]; then
                        echo -n "mkdir '$basedir': "
                        mkdir -p "$basedir"
                        chown "$user" "$basedir"
                        chgrp "$user" "$basedir"
                    fi
                fi

                if status milter-regex > /dev/null; then
                    echo -n "Already running!"
                    RETVAL=0
                    failure
                else
                    if [ -S "$socket" ]; then
                        echo -n "clean stale socket: "
                        rm -f "$socket"
                    fi

                    daemon /usr/bin/milter-regex $OPTIONS
                    RETVAL=$?
                    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/milter-regex
                fi
        fi;
        echo
        return $RETVAL
}

stop() {
        echo -n $"Stopping $prog: "
        if [ $UID -ne 0 ]; then
                RETVAL=1
                failure
        else
                killproc /usr/bin/milter-regex
                RETVAL=$?
                [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/milter-regex
        fi;
        echo
        return $RETVAL
}

restart(){
        stop
        start
}

condrestart(){
    [ -e /var/lock/subsys/milter-regex ] && restart
    return 0
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  condrestart)
        condrestart
        ;;
  status)
        status milter-regex
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart}"
        RETVAL=1
esac

exit $RETVAL

I also had to install a /etc/mail/milter-regex.conf file. Here’s a simple one that verifies the hostname in the HELO connection:

# /etc/milter-regex.conf

accept
connect // /127.0.0.1/

# Note that I modified this so that the HELO must have one dot in the hostname
tempfail "Sender IP address not resolving"
connect /\[.*\..*\]/ //
#
reject "Malformed HELO (not a domain, no dot)"
helo /\./n
#
#
reject "Spoofed HELO (my own IP address, nice try)"
helo /127\.0\.0\.1/

# It's useful to be able to add special blocks to prevent topic spam
reject "rolex spam Refused"
body /rolex/i
header /^Subject/ /rolex/i

Next, you need to add this filter into sendmail by adding the following to your sendmail.mc file

INPUT_MAIL_FILTER(`milter-regex',`S=unix:/var/spool/milter-regex/sock, T=S:30s;R:2m')
define(`confINPUT_MAIL_FILTERS', milter-regex)

Restart sendmail. When this works, you will see log messages like

sendmail[8238]: rA3FeOpB008238: milter-regex, action=helo, reject=451 4.7.1 Sender IP address not resolving
or
sendmail[8238]: rA3FeOpB008238: Milter: helo=37.212.83.52, reject=451 4.7.1 Sender IP address not resolving

Next steps

There are some more steps that are necessary. You need to set up SPF, and DKIM, and this was easy using Amazon’s tools.

You will also want to setup DMARC. I created a DNS entry “_DMARC”

which contains

v=DMARC1; p=none; rua=mailto:abuse-rua@grymoire.com;ruf=mailto:abuse-ruf@grymoire.com

This link provides more info.

I will modify this and start quarantining and rejecting mail as time goes on.

Comment

I look forward to your commands and questions. I will try to update this as time goes on.

Advertisements
This entry was posted in Linux, System Administration and tagged , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s