I work in the field of high volume mail delivery and I get a lot of questions about how email actually works. Actually, more often I am presented with a statement that is inaccurate and have to correct someone who thinks they know how it works. So to save myself from more ad-hoc explanation and to help enlighten anyone who cares, here is a whole series of "papers" on how email works and things you can do with it if you know how.
POP and IMAP for end users
When you sit down to check your mail, you most likely either log into Hotmail or Yahoo or you may open your Zimbra or Outlook mail reader. These are usually (but not always) samples of accessing email with IMAP or POP respectively. Both are systems where a server you or someone you trust manages a set of mailboxes that are accessed for reading by a specific user account. For example, bob@yahoo.com will be a specific directory on a server located at yahoo.com that only "bob" has access to read. Corporate mail systems work the same way, but the mail boxes (folders, directories) will be located on the company's own servers.
When you use a web browser to access your mail, you are most likely using IMAP (Internet Message Access Protocol) and when you use a reader like Zimbra or Outlook, you are likely using POP or Post Office Protocol. Either way, you will have a set of "folders" which represent directories on the server in your user space that you have access to. When you select "leave messages on server" in your settings, all the messages that you download to your computer to read are still kept at the server as well. This means that you can retrieve them with a different computer later if you need to, but also means that you take up more space on the server and most mail services have space restrictions you need to pay attention to. When you run out of space, senders will get a bounce message that your mailbox is full and you will not receive any more mail.
IMAP is the normal mail protocol for ISPs (Internet Service Providers) like Yahoo, Hotmail, Cox, Comcast, Earthlink, Shaw, Telus, Bell, etc, who provide access through a web interface. These also typically allow you to use a mail reader like Outlook to access them and you can often use POP to gather your mail as well. Corporate email is almost always POP but can also be IMAP. The important thing to remember is that POP and IMAP are protocols that are intended to be accessed by a human being who is reading and sending email. There are usually different folder options and view preferences that can be associated with a user account to help you better manage your email.
SMTP - the cargo train
The end user protocols POP and IMAP would be very barren places if there were no way to get email between them. If mary@hotmail.com wants to send an email to bob@yahoo.com, the email is first created in her mail reader or IMAP account, then she presses the SEND button. What happens next is all about SMTP (Simple Mail Transfer Protocol), DNS ( Domain Name System), and MTAs (Mail Transport Agents). WOW - that is a lot of acronyms - get used to it, this is an industry of acronyms.
SMTP is a protocol for delivering messages from one MTA server to another. Taking the example above, when Mary hits the SEND button, the server she is connected to will use a Mail Transfer Agent (MTA) to turn her message into something the SMTP protocol will understand. That way the MTA at Hotmail can pass the message on to the MTA at Yahoo in a form that all MTA's understand. Yahoo can then change the message back into whatever format Bob uses for his mail reader.
SMTP really only has 2 parts - the envelope and the data. The data is then broken into 2 separate parts called the headers and the body. "Headers" and "Body" are familiar to most email users who will recognize the headers as the part where the TO and FROM are and the body is where the actual message is. It is actually a bit more complex that that, but you are on the right track.
The SMTP envelope part has one and only one MAIL FROM part. This is where the message actually originated from and is usually the MTA's address on the internet. It may be incomprehensible and that is ok because a human rarely sees it. Mary's envelope MAIL FROM address may be something like 12992j3hdnji8ej88277@bounce.e140.outbound.hotmail..com which would not be a useful address for a human to send mail to, but makes perfect sense to the MTA at Yahoo receiving the message. Yahoo will use that address to determine if the mail really did come from Hotmail and validate a "return path" in case a bounce message has to be sent back.
Also in the envelope is one or more RCPT TO addresses. These are the addresses of the people who will receive the mail. Again, this is not normally revealed to the receiver but is used by the MTA to route messages. All destination addresses go in this area so if Mary also BCC'd Bob's boss, you would also find a "bobsboss@yahoo.com" in the RCPT TO area even though Bob will never see it in his mail reader. That is the entire envelope space. The next line in what we call the SMTP conversation is DATA and is where the headers and body live.
The DATA section contains the TO, FROM, and SUBJECT lines which everyone is familiar with. These are usually readable in any mail reader but there are other lines that are possible here you may not be aware of and are typically not viewable. This first section - everything before and including the SUBJECT line is called the "Headers" and can be huge. A sender can add items here in a specific format to direct an MTA to do certain things , or allow your mail reader more options. For instance, you can add a REPLY TO header that will allow a different address to be used if anyone replies to your mail. Marketing companies who have passed all the "i'm not a spammer" tests can add specific headers to tell an MTA they are a friendly sender. Google (gmail) for instance allows legitimate bulk senders to use a "Precedence: Bulk" header to indicate non-spam bulk email. Security signatures and unsubscribe processing instructions can also be stored here so your receiving MTA know how to handle the mail under certain conditions.
After the SUBJECT line, comes the body, or what most people think of as the actual 'message'. This can be plain, readable text, or it can follow the MIME standards and be available in sections. MIME (Multipurpose Internet Mail Extensions) is a way for a single message to contain different formats of the same message as well as to allow for attachments and embedded images. When a message is sent in a proper MIME format, the end user can read a plain text version or an HTML version with images attached or in-line all depending on the settings of the receivers local mail reader.
All SMTP conversations (we will get to the 'conversation' bit in a minute) end with a single dot (period) on a single line. When an MTA sees that, it closes that message and calls it complete telling the sending MTA that the message was received in good order. That does not close the connection between the servers however, so that same connection can be used to pass hundred of other messages as well before the connection is closed.
The SMTP conversation is called such because that is really what happens. Two MTA's will 'talk' to each other and perform 'handshaking'. It goes something like this:
Hi, I am hotmail.com can I send you email?
Hi hotmail.com, I am yahoo.com, sure you can send email.
This is from user "mary" on server1.hotmail.com
OK, that sender is allowed
This is going to user "bob" on server.yahoo.com
OK, that recipient exists
Here comes the DATA….
I am waiting...
TO: BOB
FROM: Mary
REPLY TO: mary21@hotmail.com
SUBJECT: This is my subject line
This is a test message
.
OK, I am done
Thanks, I got the message OK
In reality it looks like this:
EHLO yahoo.com
250-yahoo.com says EHLO to hotmail.com
MAIL FROM: mary@hotmail.com
250 MAIL FROM accepted
RCPT TO: bob@yahoo.com
250 RCPT TO accepted
RCPT TO: bobsboss@hotmail.com
250 RCPT TO accepted
DATA
354 continue. finished with "\r\n.\r\n"
FROM: mary@hotmail.com
TO: bob@yahoo.com
X-jobid: This is a special header
Subject: test
This is a test message
.
250 OK 50/00-11937-185D7EE4
quit
221 c1n1 closing connection
BOUNCES - what happens when the conversation fails
The above SMTP conversation is a perfect scenario and is what happens most of the time, but sometimes you run into errors like a bad address or a slow responding MTA. When that happens, the receiving MTA will send back a bounce. Bounces can either be In-Band (synchronous) where they happen during the SMTP conversation, or Out-of-Band (asynchronous) where they happen after the SMTP conversation has completed. If the MAIL FROM is rejected for instance, the receiving MTA will immediately return a 500 error code - often a "relaying denied" error. You may also get 500 code errors when defining a recipient that does not exist in the receiving system. In some cases the SMTP conversation is completed and some time later, the receiving MTA will discover an error (like the recipient not existing) and will return a separate message to the sender, also known as an Out Of Band bounce. In technical terms it is called a DSN (Delivery Status Notification). These are captured by the sending MTA and usually relayed back to the originate of the message.
Next time we tackle DNS - the Domain Name System
No comments:
Post a Comment