Exchange 2003 Server Service Pack 2 (SP2) Anti-Spam Framework

Here is a fantastic post by Alexander Nikolayev on the Anti-SPAM framework that is slated for E2K3 SP2...

"Spam is our e-mail customers' No. 1 complaint today, and Microsoft is innovating on many different fronts to eradicate it" - Bill Gates.

With the release of Exchange 2003 Server SP2 Microsoft and the Exchange Server Product Unit are taking another big step in helping alleviate the Unsolicited Commercial Email (spam) problem.  Announced a year ago, the Coordinated Spam Reduction Initiative clearly outlined the plan for drastically reducing the amount of spam customers may experience through the introduction of new technology solutions.  A year later, Exchange 2003 SP2 delivers exactly that - new technology that significantly helps combat spam.

With the introduction of Exchange 2003 Server, Microsoft built a framework that included a number of features intended to reduce the volume of spam.  The framework called for a solution based on a multi-layered approach around the idea that most if not all spam should be stopped at the gateway, and before reaching the final recipient's mailbox.  The adoption of this approach has proven to be very successful and highly effective, and with the release of SP2 the framework will be refined even further to include the following steps of which I'll summarize below:

1. Connection Filtering
2. SMTP Filtering
3. Content Filtering
4. Inbound mail processing rules

This is an example of typical mailflow:

At a point during mail transmission from the origination point to the final destination (recipient), a mail item enters the Exchange organization.  There it goes through the extensive anti-spam processing framework and the following anti-spam layers:

Layer 1 - Connection Filtering

The first step in spam verification process is to establish an understanding where the mail is coming from.  Highly effective, this method accounts for roughly 25% of all blocked mail inside of Microsoft Corporation.  Exchange 2003 Server SP2 provides the following framework to achieve this functionality:

1. Support for multiple Real Time Block List (also known as DNS Block List) providers (including paid subscriptions)
2. Customized rule-based Block List service configuration
3. Custom-tailored server response (DSN) based on provider and connection initiation source (i.e. open relay, known source of spam)
4. Global Accept and Deny Lists
5. Configurable exception list that override the block list

If enabled, Connection Filtering will snap the IP address on the incoming connection from the winsock and send a DNS query to verify whether the connecting identity has been listed as an open relay or if it is a known source of spam.  If the returned DNS query contains the connecting IP address on the DNS block list, Connection Filtering will close the connection down and trigger a customized server response back to the sender.  If the sender was legitimate and got onto the block list by mistake, s/he can implement corrective actions based on the custom NDR received.  Connection Filtering also allows the remote user from the blocked IP to submit mail to the postmaster or administrator of the Exchange organization for implementing corrective actions if the sending identity was put on the block list in error.  Connection Filtering is very useful as it is capable of defending Exchange deployments from spam without generating additional resource consumption (CPU cycles, further load on the downstream servers, extra mail processing and NDR handling, etc.).  Exchange 2003 Server can also perform reverse DNS lookup on incoming messages to resolve originating IP to a host name through the DNS.

The majority of Exchange 2003 Servers have been deployed behind the perimeter and do not face the Internet directly.  This renders Connection Filtering less useful as the feature relies on getting the original sender's IP to run the DNS query on.

With the release of SP2 this deficiency has been addressed by introducing a new headers parsing algorithm for originating IP retrieval.  Now, Exchange 2003 Server SP2 with Connection Filter deployed can be positioned anywhere in the organization and perform filtering as it would on the perimeter. 

Layer 2 - SMTP Filtering

If the incoming connection passed through the Connection Filtering layer, the next in line is SMTP Filtering.  Exchange 2003 Server SP2 makes extensive use of SMTP protocol filtering and this feature contributes to 35% of all blocked messages in Microsoft Corporation (MSIT department).  SMTP Filtering has been implemented to monitor live SMTP sessions and functions as a real-time filter.

SMTP Filtering has been complemented by comprehensive SMTP session RFC compliance enforcement and starts with the first RFC2821 HELO/EHLO command.  The SMTP implementation in Exchange 2003 Server SP2 will monitor and examine the session for potential violations and will not accept mail when the sending identity does not conform to or severely violates SMTP governing RFCs.  I.e. no mail transaction will be allowed if the remote party does not begin the SMTP session with an appropriately constructed greeting that contains an RFC2821 compliant domain part.  Similarly, RFC compliance will be enforced on MAIL FROM: and RCPT TO: commands to ensure that malicious users can not get around SMTP Filtering by providing, for example, 8-bit characters in the RFC2821 stream.  Another common tactic deployed by spam senders is to confuse anti-spam solutions by changing the order of commands in an SMTP session.  Exchange 2003 SP2 enforces RFC compliance in this area, preventing spam senders from executing an attack this way.

To prevent dictionary attacks and valid e-mail address harvesting, Exchange 2003 Server SP2 will respond to the VRFY command, but will not release the directory information to the remote party.  Exchange 2003 Server SP2 by default does not support the EXPN command.

SMTP Filtering is based on rules configured by the administrator and consists of Sender Filtering and Recipient Filtering.  Sender Filtering starts with the first RFC2821 MAIL FROM: command.

Sender Filtering:

1. Sender Filtering allows a list of senders to be specified that are prohibited from sending messages to a particular Exchange organization. 
2. Sender Filtering can be easily configured to reject messages originating from certain domains or email addresses. 
3. Sender Filtering filters messages with blank sender information and provides a mechanism for spoof detection (e.g. if the message coming from outside the organization claims to be sent from the CEO of organization where Exchange is deployed). 
4. Based on the administrator-configurable actions the filter can drop the incoming connection if the Sender's address matches the filter.
5. To minimize information disclosure to malicious users, Sender Filtering can silently accept mail and delete it without notifying the sender.
6. Sender Filtering provides an option for archiving filtered messages for a forensic analysis as needed. 

Once a mail item gets through Sender Filtering it faces  Recipient Filtering.  Recipient Filtering starts with the first RFC2821 RCPT TO: command.

Recipient Filtering:

1. Recipient Filtering enables inbound mail filtering for a particular recipient in the Exchange organization. 
2. Recipient Filtering supports blocking mail based on wildcards.  This enables administrators to use patterns to block entire ranges of recipients.
3. Recipient Filtering filters messages sent to non-existent recipients, rejecting them at the protocol level.  By rejecting non- existent recipients at the protocol level (on RFC2821 RCPT TO: command), the Exchange server is protected from doing expensive NDR generation work and clogging the Badmail directory.

4. Enabling filtering of the Recipients who are not in the Directory potentially allows spam senders to discover internal directory information (valid e-mail addresses in the Exchange organization).  A malicious user can execute address book mining by monitoring/parsing the server responses to RCPT TO: commands.  To mitigate this threat Exchange 2003 Server SP2 supports SMTP command tarpitting.  An administrator can configure Exchange to implement an n-seconds delay of the server response to the RCPT TO: command if a DHA attack is encountered or if the remote party violates SMTP RFC conformance.  When a malicious user tries to harvest responses the Exchange server significantly slows down its responses (to an admin-defined delay interval) and the attack becomes infeasible. 
5. Ability to restrict Distribution List mail submissions to authenticated users only contributes to the Recipient Filtering Framework.
6. Recipient Filtering applies only to anonymous connections so all authenticated identities bypass Recipient Filtering rules. 

Layer 3 - Content Filtering

If a mail item gets through Recipient Filtering it faces Content Filtering.  Content Filtering in Exchange relies on Microsoft Research SmartScreen machine learning technology incorporated into the Intelligent Message Filter (IMF).

Intelligent Message Filter:

Messages from the Internet arrive at the Exchange SMTP gateway and enter the Exchange 2003 Server anti-spam framework. Previous layers of the Exchange 2003 SP2 anti-spam solution (connection, sender and recipient filtering) block message submission before message data is sent.  If a message passes all of these then the message body is received. A custom event sink (msgfilter.dll) is invoked when the SMTP End of Data event occurs. The sink passes the message to the Intelligent Message Filter SmartScreen DLL. The SmartScreen technology determines the Spam Confidence Level (SCL) rating of the message which is returned to the sink for comparison against the gateway SCL threshold. The Administrator defined action is applied if the message SCL is greater than the gateway threshold.  Otherwise, the SCL rating is added to messages for transmission to the recipient's inbox.

As of today, the SmartScreen technology deployed by the Microsoft IT department blocks 40% of remaining messages that passed through the previous anti-spam layers (Connection and SMTP filtering).

Anti-Phishing:

With the introduction of Exchange 2003 Server SP2 Intelligent Message Filter extends the anti-spam functionality to support anti-phishing.  The new SmartScreen anti-phishing technology will impact the SCL ratings and also expose an independent Phishing Confidence Level (PCL) value as an output of the filter.  The new anti-phishing functionality is totally transparent to administors and includes the following features:

1. Anti-phishing heuristics
2. Anti-phishing consistency list
3. Anti-phishing block list
4. Anti-phishing allow list
5. PCL values map to weights within the DAT file that allow for non-deterministic SCL adjustment  

Anti-phishing technology relies on heuristics, the consistency list, and the allow list to detect phishing scams.  These lists are not exposed for administration and are updated during regular IMF updates (frequency of updates will be determined later).  The PCL score is one of the factors that trigger final SCL assignments. 

Custom Message Weight:

Custom Weighting (also known as Bad Words List) is a file-based implementation with no supporting UI.  Within the file specific words/phrases can be added along with their relative text part location (Subject or Body) and their associated modifier value.  The supported modifier values can include positive and negative increments, as well as MAX and MIN values.  During anti-spam mail processing the IMF will look into the file and search inside the mail item for a string/word/phrase match.  If a match is found the weight of the SCL will be adjusted according to the modifier value.  If MAX or MIN values are specified for particular word or phrase, they will move the SCL towards MAX or MIN.  The Custom Message Weight feature allows administrators to custom tailor the filter to account for specific words or phrases and adjust it on the fly to act on particular message content as needed.

Custom Server Response when IMF is configured in 'Reject' mode:

To mitigate False Positives, a problem common to all types of automated anti-spam technologies, IMF can be configured to work in "Reject" mode.  This allows for False Positives investigations if the mail was submitted by a legitimate sender and rejected by the filter.  SP2 allows customization of the server response string that is generated and appended to the NDR sent back to the sender.  The default response "550 5.7.1 Requested action not taken: message refused" can be altered to provide blocked legitimate users with meaningful explanation as to why their message was blocked.  This will alleviate investigation of False Positives and decrease support costs associated with the anti-spam processing.

Sender ID Filtering:

Sender ID is an industry standard framework created to counter e-mail domain spoofing.  Sender ID is aimed at removing the ambiguity associated with the sender identity by verifying that each e-mail message originates from the Internet domain from which it claims to come based on the sending server's IP address. Eliminating domain spoofing will help legitimate senders protect their domain names and reputations, and help recipients more effectively identify and filter junk e-mail and phishing scams.

The steps in the process are:

1. The Sender sends an e-mail message to the Receiver.
2. The Receiver's inbound mail server receives the mail.
3. The Receiver's server checks for the SPF record of the sending domain published in the Domain Name System (DNS) record.
4. The Receiver's e-mail server determines if the sending e-mail server's IP address matches the IP address that is published in the DNS record.

Sender ID defines an algorithm for detecting the email address of the entity that is most recently responsible for injecting a message into the email system by extracting the Purported Responsible Address (PRA). The extraction of the PRA ensures Sender ID verifies the appropriate sender against the correct IP addresses as email systems can legitimately forward mail on behalf of other mail servers.

The Sender ID feature has 3 modes:

1. Delete (silent delete - no NDR generated)
2. Reject (the mail will be rejected at the protocol level)
3. Accept (the mail item will be stamped with the Sender ID result for IMF consumption). 

The first and second mode will delete or reject mail that failed the Sender ID verification (i.e. a clear case of spoofing), the rest of the mail items will be stamped with the Sender ID status and passed along.  The last option will just stamp the Sender ID status onto the mail item (even in the case of spoofing).  This status will be passed to the new Intelligent Message Filter and trigger appropriate Spam Confidence Level (SCL) score modification.