SMTP

From Postmaster Administration Wiki
Jump to: navigation, search

Simple Mail Transfer Protocol

Firewall-SMTP.png

Mail is sent between network-connected hosts using SMTP, the Simple Mail Transfer Protocol. A mail exchange (MX) is an Internet-facing host that accepts SMTP connections on port 25 and/or the SSL port 465, and may relay mail to other Internet connected hosts via port 25.

In an organisation's network firewall, port 25 is open only for inbound mail traffic to MX hosts and for outbound mail hosts to relay mail. Port 25 should otherwise be closed to all other network traffic to prevent virus/malware infected machines from abusing the mail system. The mail submission port 587 exists for handling authenticated submissions of new mail.


Protocol Summary

SMTP is a synchronous request-reply human readable protocol. It is state oriented, which means that commands have to be applied in a specific order and/or state. Successful commands return 250 or similar 200 series code; temporary errors return a 400 series code, and permanent errors or rejections return a 500 series code. See below for a more detailed explanation of SMTP replies.

Using a telnet client one can connect to port 25 of a mail server and get a "220 mx.example.com text" welcome banner, after which the following commands can be used starting with either EHLO or the older HELO.

EHLO fqdn

EHLO was introduced by RFC 2821 as means to discover server capabilities and extensions. It should be the first command issued after a connection is established and before a MAIL transaction and/or after STARTTLS. The fqdn must be the connecting client's host name or an IP-domain-literal. While not RFC conformant, many MTAs accept bare IP addresses, i.e., missing the square brackets. Note that specifying localhost, [::1], or [127.0.0.1] for a connection that does not originate from the loopback interface will probably be rejected and/or dropped as spam behaviour. EHLO can be issued at any time, except during message content transfer state, and behaves like RSET.

STARTTLS

A SMTP connection on port 25 starts unencrypted. This command requests that the server start TLS, communication channel encryption. Some SMTP servers require TLS before accepting clear-text user authentication using AUTH PLAIN or AUTH LOGIN. A client should issue an EHLO immediately following STARTTLS as the list of available capabilities and extensions may change once encryption is enabled. It is not possible to test STARTTLS using telnet, see Manually Test STARTTLS.

AUTH method [argument]

SMTP AUTH is an extension that allows for assorted mechanisms to be used to accept authentication credentials, i.e., account name and password details. Most common methods are LOGIN (used by Microsoft products) and PLAIN, which are insecure clear text mechanisms support by all MTA and MUA. Some other AUTH mechanisms include CRAM-MD5, DIGEST-MD5, and KERBEROS V5, and support for them varies.

An SMTP session does not necessarily require authentication to deliver mail. Authentication is typically used to verify trusted servers or senders so as to relay mail through the server and/or bypass anti-spam filters. However, the need for this facility has been largely replaced by the mail submission agent on port 587, which is the preferred means to introduce new mail into the mail system, since many Internet service providers block port 25 outbound.

When testing AUTH PLAIN manually using telnet or openssl s_client, the username and password need to be Base64 encoded. This can be generated from the command line then cut/paste or scripted into the SMTP session. For example:
$ printf "\0username here\0password here" | openssl base64
AHVzZXJuYW1lIGhlcmUAcGFzc3dvcmQgaGVyZQ==
$

NOTE though seldom used, AUTH PLAIN supports a realm argument that can be placed before the first \0, eg.

$ printf "realm here\0username here\0password here" | openssl base64
cmVhbG0gaGVyZQB1c2VybmFtZSBoZXJlAHBhc3N3b3JkIGhlcmU=
$

HELO fqdn

HELO is the original RFC 821 command by which a system "introduced" itself (reasons for which are unclear). It should be the first command issued before the first MAIL transaction. Its use has been deprecated and signals original SMTP semantics, so no extensions, such as PIPELINING or STARTTLS. HELO can be issued at any time, except during message content transfer state, and behaves like RSET.

MAIL FROM:<sender> [parameters...]

Starts a mail transaction, which ends when either the end of message dot is sent or RSET is given prior to DATA. The sender can be the empty string, e.g. MAIL FROM:<>, which is referred to as the "null" sender, and is used for reporting mail delivery errors (DSN) and status (MDN). Otherwise the sender is a qualified mail address, i.e. local-part@domain.tld. There can only be one sender per MAIL transaction.

RCPT TO:<recipient> [parameters...]

There can be one or more mail recipients specified per MAIL transaction. The null address is not valid. It is possible to specify an unqualified recipient, i.e., without the @domain.tld portion, but only the unqualified postmaster address is required to be supported. The handling of unqualified recipients, other than postmaster, varies according to local policy.

DATA

End the list of recipients and enter the message content transfer state. The DATA command normally replies with a 354 go-ahead code or a temporary or permanent error. Message data is sent in line units of 1000 bytes or less including the CRLF newline. The end of message is given by sending a line containing only a period (.) and CRLF, after which the server will send a reply accepting or rejecting the message. Message lines that start with a period (.) must be "dot stuffed", essentially an extra dot prepended to the line.

RSET

Abort a MAIL transaction, discarding the sender and any accepted recipients. This command can be issued at any time, except during message content transfer state.

NOOP

Do nothing. Used to force a sync. point for PIPELINING and/or keep a connection from timing out. This command can be issued at any time, except during message content transfer state.

QUIT

Say good night Gracey. Sends a 221 reply line and disconnects. This command can be issued at any time, except during message content transfer state.

ETRN fqdn

Used to signal that the client is ready to receive queued mail for fqdn via separate SMTP connections from the server. Cannot be issued during a MAIL transaction.


Each successful MAIL command during the SMTP session starts a new message transaction, which ends when either the end of message dot is sent or RSET is given prior to DATA. For each message, there can be more than one RCPT given. For each SMTP session, many mail transactions can be made, in theory an unlimited number.

Of the information obtained from each state, only the IP address of the connected SMTP client and each valid RCPT address specified can be relied upon. Even then, the connecting IP might be questionable, because it's possibly in a dynamic IP address pool, the reverse DNS of the IP is often poorly configured or non-existent, or the WhoIs information about IP and domain assignment might be restricted, due to privacy concerns (RFC 3912). As for the other states, the HELO, MAIL, and message content can be misrepresented or faked, and thus cannot be immediately trusted.

Most spam filtering techniques fall into two classes: those that act on the SMTP client's IP address and envelope information (pre-DATA) and those that act on the message content (post-DATA). This distinction is important, because once the DATA command is accepted by the receiving server, it is generally committed to reading the entire message until the SMTP client indicates it has finished. This, of course, consumes bandwidth and system resources, so there are filtering techniques that attempt to make a decision based on policy or behaviour before accepting DATA in order to avoid/reduce more expensive forms of content filtering after.


SMTP Reply Codes

The SMTP reply codes are three-digit codes that appear at the start of a SMTP response followed by human readable text. The codes are used by software to determine protocol state; the human readable text has no meaning to mail software and is not standardised, so the text can be anything from an empty string to verbose descriptions. SMTP replies can be one or more lines; a hyphen ( - ) following the reply code indicates more lines to follow, and a reply code followed by an ASCII space character is the end of a multiline reply. Multiline replies are common for the welcome banner and EHLO reponse, but can occur in response to any command.

Multiline Welcome Banner Example

220-mx.example.com HI! #633 p47FEY313020014000
220-I'm Eddie your onboard computer!
220-I'll be very happy to handle your mail transactions today.
220 Won't this be fun! Afterwards maybe we can do lunch?

The three digits (xyz) of a reply code have significance. The first digit ( x ) denotes the successful disposition or not of the command, the second digit ( y ) the category, and the third digit ( z ) supplemental information. There are four possibly disposition classes:

  • 2yz for positive successful results;
  • 3yz for intermediate, in progress, commands;
  • 4yz for temporary error conditions;
  • 5yz for permanent errors or rejections.

The categories are:

  • x0z for syntax related;
  • x1z for informational;
  • x2z for connection details;
  • x3z more input expected;
  • x4z currently not used;
  • x5z for mail system.

So in the welcome banner example above, a 220 indicates a successful connection response, a 421 indicates busy, come back later, and a 554 indicates no service (though one would have expected 524 given the definitions above).

The most common codes seen during a SMTP conversation are:

  • 220 welcome
  • 221 good-bye
  • 214 help information
  • 250 ok
  • 334 more input for AUTH
  • 354 more input for DATA
  • 421 service busy, halting, closing connection
  • 450 mailbox busy or temporary security policy block
  • 451 aborted, "try again later"" (used by grey-listing)
  • 500 command syntax error such as unknown command or line too long
  • 501 argument or parameter syntax error
  • 502 command not implemented
  • 503 bad command sequence, command out of expected order, wrong state
  • 504 parameter not implemented
  • 550 action failed or rejected
  • 553 mailbox or address error
  • 554 transaction failed; "no service here" when used for a welcome banner

There are several other less frequently seen reply codes; see the RFC for the complete list. There exists a SMTP extension for enhanced reply codes, which attempts to provide finer and specific details. When available, the extended reply codes are a dot separated tuple that follows immediately after the reply code and hyphen or space delimiter.


Enhanced Reply Code Example.

MAIL FROM:<>
250 2.1.0 sender <> accepted #283 p47GCM313020418000

SMTP Timeouts

RFC 5321 section 4.5.3.2 covers the client and server timeout values for various SMTP command states. A postmaster should not change their [Agents.html MTA]'s default timeouts from the RFC historical values. While the timeout values are rather lengthy, there is a good reason not to change them. The values originally allowed for network connection over modems, which operate much slower than today's modern ADSL and fibre optic networks. While one would think these timeouts could be scaled to match today's network speeds, this would be a bad thing to do. Network speeds have increased, but so too has the volume and size of network traffic.

A small mail system may have to handle 100s of messages per minute, because of junk mail and malware, while high-volume mail systems for a university or ISP may have to handle 100,000s of messages per minute. SMTP connections can last seconds to several minutes, even hours. Consider the network speeds of some countries, like South Africa, whose infrastructure has not yet caught up with that of North America, Europe, or Japan. Also many mail systems implement anti-spam and anti-virus filtering, which may take advantage of SMTP timeouts in their operation.

Do NOT alter the SMTP protocol level timeouts as it can have undesired effects with respect to message delivery.

RFC 5321 section 4.5.4 covers retry and sending strategies, i.e., what to do if a message doesn't get through on the first attempt. Most MTAs have at least a queue retry interval and message expire timeout; some MTAs, like Sendmail, have finer granularity of retry and expire values based on message precedence, e.g., 1st class, mailing-list, bulk, junk, or similar criteria.

The queue retry interval tells the MTA how long to wait between delivery attempts. The RFC recommends that this value be at least 30 minutes. Some postmasters increase this value, while more often than not, many postmasters make the mistake of shortening this value to an absurdly small number, sometimes on the order of seconds. It is recommended that a queue retry be no smaller than five (5) minutes. Anything smaller many have adverse effects on your mail system and/or result in temporary or permanent blacklisting actions on the part of receiving mail servers.

Too short a queue retry interval will probably result in thrashing the outbound mail system, placing excessive load on CPU and disks, if the number of messages in the queue is high. Constantly reconnecting every few seconds in attempt to deliver a message, sometimes referred to as "wood peckering", will not improve one's ability to successfully deliver a message, especially when a receiving MX uses a grey-listing anti-spam strategy and will more likely result in your system being blacklisted.

Some queue retry intervals use a non-linear strategy, such as an "exponential backoff", where the retry interval starts small, like one minute, and is doubled after each failed attempt to deliver a message until the message finally is expired from the queue.

Do NOT specify a queue retry interval less than five (5) minutes.

The message expire timeout is how long a message will remain in the message queue until it is expired and an error sent to the sender. Historically the message expire timeout was five days, to account for older UUCP based mail deliver. However, with today's Internet most users assume mail delivery on the order of minutes. While the message expire timeout can be shortened, it is recommended that a value no smaller than three (3) days be used, i.e., a long weekend. Most mail systems operate unattended, but there can still be network or server outages for hours or even days. A good rule of thumb for setting the message expiry timeout is that it should correspond to the longest period of time that the mail system might be unattended; for a small business, that would probably be the equivalent of a three day long weekend.

Do not specify a message expire timeout shorter than three (3) days.


References

  • RFC 5321 Simple Mail Transfer Protocol (previously RFC 2821, 821)
  • RFC 1870 SMTP Service Extension for Message Size Declaration (SIZE)
  • RFC 1985 SMTP Service Extension for Remote Message Queue Starting (ETRN)
  • RFC 2076 Common Internet Message Headers
  • RFC 2195 IMAP/POP AUTHorize Extension for Simple Challenge/Response (CRAM-MD5)
  • RFC 2831 Using Digest Authentication as a SASL Mechanism (DIGEST-MD5)
  • RFC 2920 SMTP Service Extension for Command Pipelinig (PIPELINING)
  • RFC 3207 SMTP Service Extension for Secure SMTP over Transport Layer Security (STARTTLS)
  • RFC 3461 Simple Mail Transfer Protocol (SMTP) Service Extension for Delivery Status Notifications (DSN)
  • RFC 3463 SMTP Enhanced Mail System Status Codes, see RFC 5248
  • RFC 3464 An Extensible Message Format for Delivery Status Notifications (DSN format)
  • RFC 3798 Message Disposition Notification (MDN format)
  • RFC 3848 ESMTP and LMTP Transmission Types Registration
  • RFC 4616 The PLAIN Simple Authentication and Security Layer (SASL) Mechanism (PLAIN)
  • RFC 4752 The Kerberos V5 ("GSSAPI") Simple Authentication and Security Layer (SASL) Mechanism (GSSAPI)
  • RFC 4954 SMTP Service Extension for Authentication (AUTH) (previously RFC 2554)
  • RFC 5248 A Registry for SMTP Enhanced Mail System Status Codes
  • RFC 5322 Internet Message Format (previously RFC 2822, 822)
  • IANA SMTP Enhanced Status Codes Registry
  • Wikiperdia Bounce Messages Explained