Email system part #1, the main components.
A Quick Overview.
Introduction
One of the most important and useful applications of the internet is undoubtedly the sending and receiving of emails. Apart from their almost instant nature they are also effectively free. Unfortunately this also makes email an ideal medium of abuse, in the sense that because it is free to send an email, it is also free to send a million emails. The major ISP’s estimate that of the billions of emails being sent and receiving, about 50% of them are spam (or UCE - Unsolicited Commercial Email). For many people who have the same email address for a period of years, the amount of spam can be counted in thousands a day. By the end of 2003 I was receiving 200-300 per day.
So, email is important, and spam is a nuisance that slowly effects all users. As a Site Manager, you will need to understand how to run a secure email server and also how to setup filters for spam. The first step is to understand how email moves through our system.
Before that I should note that the way Westhost installs the email programs is probably fine for the majority of users. It is secure and it works. What follows if further information for folks who really have a problem with spam, or who are simply curious on how the whole email thingy works. They also have a set of help pages in the Email section of the Westhost Manual.
Although email seems to be a simple system, there is a lot going on underneath and a variety of different components involved in the process. The rest of this page is a quick (and greaty simplified) overview of the process.
When alice@somedomain.com sends an email, lets say to joe@mydomain.com, the sending machine (somedomain.com) asks at port 25 of the server that runs mydomain.com to handle the mail. The sendmail program on mydomain.com is started up and somedomain.com asks if it will handle an email for joe@mydomain.com. Sendmail then looks at its users and access lists and decides whether or not it should accept the email. If it doesn’t accept the email it will respond with an error code and a reason and terminate the connection.
If it does accept the email it first checks if the user has a forwarding file (/etc/virt_user/username/.forward) and, if there is a forwarding address, forwards it to that address. If not, it looks in its ruleset to find how to process the email. Although sendmail is quite capable of delivering the email itself to the users mail directory, on Westhost sendmail is told to send the email off to procmail to deal with.
procmail in turn has a set of rules (in ~/.procmailrc) that tell it how to handle the email. One of those rules could be to send the email off to spamassassin to check.
spamassassin in turn has another set of rules and filters which it checks the email against. It then returns the email to procmail (after modifying the headers if necessary) and procmail continues with delivery.
That, in a nutshell, is the approximate process that the email passes through. Quite a lot of rules and filters on the way and plenty of opportunity to stop unwanted email and allow through the legitimate stuff.
From the above it should be clear that we have three places to check and deal with the emails:
- sendmail
- procmail
- spamassassin
Let’s look at each of those as an overview, I’ll cover them in more detail on the next page.
Sendmail
This is, in theory, the best place to catch the unwanted emails. It’s the best place because we don’t even need to download the email, just refuse it from the start. I say ‘in theory’ because sendmail has one drawback and that is that it is very complex to setup and one mistake can create a situation where all emails are lost. Plus, if we reject the email from the outset, we have no way to check whether it really was spam or not. So we are in practise a little limited in what we can ask sendmail to do for us.
In short, what sendmail can do is:
- refuse email from certain domains or accounts
- refuse email sent to certain domains or accounts
- check with an online blacklist (RBL) if the from address is known for spam.
Refusing email from certain domains and/or checking with RBL’s sounds excellent, but in practise doesn’t work so well. Most spam is relayed through temporary ‘open relays’ and will not immediately appear on an RBL. And the domain could be something as generic as ‘comcast.net’ or ‘rr.com’ (both major sources of spam BTW) which usually indicates that it is a normal user whose comuter has been ‘trojaned’ and is being used to send spam.
Procmail
Basically, procmail takes the email and matches it against a set of rules defined in the .procmailrc file. The rules can be set to match against any of the headers or against the body of the email. If there is a match, then an action can be specified, such as deliver the email, send to spamassassin or delete it.
SpamAssassin
spamassassin also matches the email against a set of rules, the difference being with spamassassin that there are many rules supplied with the installation that take care of the hard work for you. Spamassassin also has the ability to look up the email against a RBL. The program knows nothing of mail delivery though, and just returns the email to procmail to continue with, after rewriting the headers of the email to indicate the spam score.
With both procmail and spamassassin the email has already been accepted, although procmail has the ability to tell sendmail to bounce the email if required. This is generally not a good idea, as the rejection will possibly be sent to a faked email address, in which case it will bounce back to you, thus increasing the email traffic with bounce rejections.