Direct Push is just a heartbeat away

Another good explanation of Direct Push posted at EHLO… 

Background   Exchange 2003 introduced the Always Up To Date notification feature (AUTD) that kept devices up to date by sending SMS triggers to the device. The triggers were sent from the enterprise as SMTP messages to the SMTP front end at the mobile operator. They were then sent through the SMS gateway as SMS messages to the device. This approach had some limitations since not all mobile operators did the SMTP to SMS conversion. Even when they did, there was latency involved with SMS messages and there were end-to-end reliability issues. Also some mobile operators charged for each incoming SMS message so that added an extra dimension to the cost of staying up to date. To alleviate these issues, Exchange 2003 SP2 introduced Direct Push.   Direct Push Architecture   Direct Push is a client initiated HTTP connection to the server where the device opens a connection to the Exchange Server and keeps it alive for a duration known as the heartbeat interval.  Basically the client sets up the connection, chooses the appropriate heartbeat interval and tears down and reestablishes the connection if and when necessary. The server sends notifications about new items over this connection and the client synchronizes to get the new items.   A new AirSync command called PING has been introduced for Direct Push. This command is sent as part of the POST request from the device. Summary of Interaction between the client, EAS server and Exchange 1. Device issues a PING command. 2. When the EAS server receives a PING command it does the following: ·         If the Ping command contains the heartbeat interval or folder list, it stores the information in AUTDSTATE.XML in the user's mailbox. The device does not need to send these parameters up again unless they change. ·         If the Ping command did not contain the heartbeat or folder list, it retrieves them from the mailbox server. ·         EAS subscribes to notifications for the folders. It issues DAV subscriptions using the SUBSCRIBE command. ·         Since there is a small window between the last SYNC and the SUBSCRIBE where changes could have occurred, EAS checks for changes. If there is a change, the server immediately notifies the client to sync by issuing a response to the PING command with a Status of 2. It does an UNSUBSCRIBE to delete the DAV subscription. If no changes have occurred, the server continues to wait for UDP notifications from the mailbox server. ·         If a notification arrives within the heartbeat interval, the server will inform the client to sync. A response to the PING command is issued with a Status of 2 indicating that there are changes. Otherwise, after the heartbeat interval elapses, the server will return a response to the PING command with a Status of 1 indicating that there are no changes. It does an UNSUBSCRIBE to delete the DAV subscriptions before issuing the PING response.   Deployment Considerations for Direct Push   1. In order to use Direct Push, only the Exchange 2003 Front End servers need to be upgraded to SP2. However it is highly recommended that SP2 be installed on all Exchange Front End and back end servers.    If the Front End servers are load balanced, all the Front End servers need to be upgraded around the same time.   2. When there is new mail, the BE sends a UDP notification to the FE.  Direct Push requires that UDP port 2883 be open from the BE to the FE. The port can be configured using the registry value UDPListenPort under HKLM\SYSTEM\CurrentControlSet\Services\MasSync\Parameters. If this value is set through the registry, the value must be greater than or equal to 1 and less than or equal to 65535.    3. With Direct Push, the device keeps a connection open to the Exchange server. If you have a firewall between the device and the Exchange server, you must increase the idle connection timeout on the firewall. Please note that this is the idle connection timeout (i.e.) when there is no data transfer between client and server. For more information, please refer to KB titled "Enterprise firewall configuration for Exchange ActiveSync Direct Push Technology" available at http://support.microsoft.com/?kbid=905013   4. If you are using ISA 2000, you need to add a registry key on the ISA server to use direct push. Please refer to  the KB titled "The ISA Server response to client options requests is limited to a predefined" available at http://support.microsoft.com/?ID=304340 for information on how to add the registry key.   Heartbeat Interval   The device specifies the heartbeat interval as part of the PING command. This dictates how long the server must keep the connection alive. The device will dynamically converge to the highest possible heartbeat interval for a given network, based on the mobile operator timeouts, firewall timeouts etc. The higher the heartbeat interval, the better it is for battery life. So the heartbeat is optimized for a given network.   You can change the minimum and maximum heartbeat interval settings on the server through the registry.   The settings are MinHeartbeatInterval and MaxHeartbeatInterval under HKLM\SYSTEM\CurrentControlSet\Services\MasSync\Parameters   The defaults are 1 and 45 minutes respectively. Note that the maximum is hard coded to 59 minutes since the maximum possible DAV subscription lifetime is 60 minutes.   You can also specify a heartbeat alert threshold. The server maintains a sliding window of the last 200 heartbeat intervals supplied by clients. If the average from this sample is less than or equal to the alert threshold, there will be a warning in the event log     "The average of the most recent heartbeat intervals used by clients is less than or equal to x. Please check your firewall settings to ensure that they permit requests to Exchange ActiveSync to live for at least 15 minutes."   The alert threshold and sample size can be configured through the registry. The settings are HBiSampleSize and HbiAlertThreshold under HKLM\SYSTEM\CurrentControlSet\Services\MasSync\Parameters   Configuring Direct Push on the Server   By default, Direct Push is enabled in Exchange 2003 SP2. However you can enable/disable it in Exchange System Manager. In ESM expand Global Settings, right-click on Mobile Services, Properties and check/uncheck the box for "Enable Direct Push over HTTP(S)"     You can also change this setting on a per-user basis using Active Directory Users and Computers.  In ADU&C, click on the user, Properties, Exchange Features tab, under Mobile Services enable/disable Up-to-Date Notifications. This controls both SMS based AUTD and Direct Push for the user.   Configuring Direct Push on the client   A Direct Push capable device will automatically negotiate the protocol with the server and configure itself to use Direct Push. The sync schedule is set to "As new items arrive".   Direct Push Initialization   1. Verify that Exchange ActiveSync is loaded and IP-based AUTD is initialized by checking the application log on the FE for events below. Exchange Activesync gets initialized on the first sync attempt.   Event Type: Information Event Source:     Server ActiveSync Event Category:   None Event ID:   3002 Date:       3/19/2006 Time:       12:44:08 PM User:       N/A Computer:   1B25A Description: Microsoft Exchange ActiveSync has been loaded: Process ID: [3048].   Event Type: Information Event Source:     Server ActiveSync Event Category:   None Event ID:   3025 Date:       3/19/2006 Time:       12:44:19 PM User:       N/A Computer:   1B25A Description: IP-based AUTD has been initialized.   2. Verify that the FE is listening on port 2883.   To check if the server is listening on the AUTD port, you can run "netstat -ano". Here are results before and after IP-based AUTD has initialized.   Before   Proto       Local Address     Foreign Address   State       PID   UDP         0.0.0.0:1985      *:*                           1928 UDP         0.0.0.0:3456      *:*                           3356   After   Proto       Local Address     Foreign Address   State       PID   UDP         0.0.0.0:1985      *:*                           1928 UDP         0.0.0.0:2883      *:*                           3048 UDP         0.0.0.0:3456      *:*                           3356   Netstat provides the Process ID which matches the EAS process per the initialization event in the application log.   Another way to check if the server is listening on the AUTD port is to use PortQry(available on Microsoft.com). This lists the process that is listening on the port   Process ID: 3048 (w3wp.exe)   PID   Port        Local IP          State             Remote IP:Port 3048  TCP 31479  172.29.8.222      ESTABLISHED       172.29.9.107:3268 3048  TCP 31480  172.29.8.222      ESTABLISHED       172.29.9.107:389 3048  UDP 2883    0.0.0.0                             *:*   Troubleshooting using logs   1. Enable device side logging. The logs are saved in text format in the Windows\ActiveSync folder. PING commands will be logged in "Ping Exchange Server x.txt" where x =1,2,3.  You should see commands similar to the one below.   POST Microsoft-Server-ActiveSync?User=administrator&DeviceId=6F24CAD599A5BF1A690246B8C68FAE8D&DeviceType=PocketPC&Cmd=Ping MS-ASProtocolVersion: 2.5   The POST command is also logged in the IIS log on the FE.   The Ctrl log on the device can also be used to troubleshoot Direct Push although the format of this file may change with device updates.   2. Check the IIS logs on the BE to see if AUTDState.XML is being created or updated. You should see an entry something similar to the one below.   PUT /exchange/Administrator@1b1domain.lab/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/PocketPC/6F24CAD599A5BF1A690246B8C68FAE8D/AutdState.xml   Note: The AUTDState.XML is created on receipt of the 1st PING request and is updated only when the heartbeat or folder list changes. So you may not see this command for every Ping request.   AUTD state information is maintained on the mailbox server in the NON_IPM_SUBTREE of each user's mailbox.    In IE, you can Choose File, Open, check the box to "Open as Web Folder" and type in http://server/exchange/user/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/Autd-State.XML   Sample AUTDState.XML   <?xml version="1.0" encoding="utf-8"?> -<AutdState xmlns="Ping:">    <Version>1.0</Version>    <HeartbeatInterval>680</HeartbeatInterval>                 -<Folders>   -<Folder>          <Id>7529a5b36290aa458b9e1fc2d5ff85a6-3aaa2</Id>       <Class>Email</Class>    </Folder>   - <Folder>     <Id>7529a5b36290aa458b9e1fc2d5ff85a6-2cfb8</Id>     <Class>Calendar</Class>     </Folder>   …  </Folders> </AutdState>   3.  Check the IIS logs on the BE to see if SUBSCRIBE commands are being issued from the FE to the BE(ie) if DAV subscriptions are being created.   For example, you should see something similar to   SUBSCRIBE /exchange/Administrator@1b1domain.lab/Inbox/   4. You can run a netmon on the FE to see if UDP notifications are being sent over port 2883 from BE to FE.   551 16.781250 LOCAL 000E0C06CAC0 UDP Src Port: Unknown (33660); Dst Port: Unknown (2883); Length = 162 (0xA2) BE FE IP   UDP: Src Port: Unknown (33660); Dst Port: Unknown (2883); Length = 162 (0xA2)     UDP: Source Port = 0x837C     UDP: Destination Port = 0x0B43     UDP: Total length = 162 (0xA2)     UDP: UDP Checksum = 0xC233     UDP: Data: Number of data bytes remaining = 154 (0x009A) 00000:  00 0E 0C 06 CA C0 00 D0 B7 24 86 2B 08 00 45 00   ....ÊÀ.÷$†+..E. 00010:  00 B6 C8 73 00 00 80 11 07 3A AC 1D 09 71 AC 1D   .¶Ès..€..:¬..q¬. 00020:  08 DE 83 7C 0B 43 00 A2 C2 33 4E 4F 54 49 46 59   .Þƒ|.C.¢Â3NOTIFY 00030:  20 68 74 74 70 75 3A 2F 2F 31 62 32 35 61 2E 31    httpu://1b25a.1 00040:  62 31 64 6F 6D 61 69 6E 2E 6C 61 62 3A 32 38 38   b1domain.lab:288 00050:  33 2F 33 35 33 39 35 63 65 34 2D 31 35 30 34 2D   3/35395ce4-1504- 00060:  34 61 63 34 2D 39 37 32 31 2D 66 31 35 32 63 36   4ac4-9721-f152c6 00070:  34 36 65 61 33 35 20 48 54 54 50 2F 31 2E 31 0D   46ea35 HTTP/1.1. 00080:  0A 53 75 62 73 63 72 69 62 65 2D 67 72 6F 75 70   .Subscribe-group 00090:  3A 20 55 73 50 43 57 77 46 4C 32 30 71 37 44 2B   : UsPCWwFL20q7D+ 000A0:  6E 61 76 6F 4D 71 79 41 3D 3D 0D 0A 53 75 62 73   navoMqyA==..Subs 000B0:  63 72 69 70 74 69 6F 6E 2D 69 64 3A 20 32 37 0D   cription-id: 27. 000C0:  0A 0D 0A 00           Frequently Asked Questions and Answers   1.    Does Direct Push work for folders other than inbox?   Yes, Direct Push is available for mail folders, Contacts, Calendar and Tasks. The list of folders for Direct Push is the same as the list of folders that have been configured for sync.   2.    What devices support Direct Push?   Windows Mobile 5 devices require the Messaging and Security Feature Pack(MSFP) for Direct Push. MSFP is included with AKU2.2. So any Windows Mobile 5 device that has AKU2.2 supports Direct Push.  The AirSync protocol has been licensed to several companies such as Palm, Motorola, Nokia, Symbian, Dataviz and SonyEricsson. Please contact the licensees to see if Direct Push capable devices are available.   3.    Is Direct Push supported over Wi-Fi?   No. Direct Push requires a cellular data connection. It is not supported over Wi-Fi or Desktop Passthrough(when the device is cradled).   Due to hardware limitations, Wi-Fi cannot go into standby mode and receive notifications. So in order to support Direct Push over Wi-Fi, the Wi-Fi connection would have to be kept alive which in turn would drain the battery very rapidly.   4.    Does Direct Push work with SecurID?   RSA has an update to their agent to allow it to work with Direct Push. RSA Authentication Agent 5.3 for Web for IIS enables you to use Exchange ActiveSync without having to reauthenticate every time ActiveSync is invoked. For more details, please read this and contact RSA.   5.    Does Direct Push have an impact on server performance?   A typical FE services several thousand connections from clients using OWA, OMA, EAS, and RPC/HTTP clients. Based on the testing done by Microsoft IT, the additional connections opened by Direct Push did not require the deployment of any additional FE or BE servers. It also did not require an upgrade of hardware on existing servers.   For more information please refer to the whitepaper titled "Microsoft IT Scalability Experience with Windows Mobile 2003 and Exchange Server 2003 Mobile Messaging" available at   http://www.microsoft.com/windowsmobile/business/strategy/scalability.mspx   - Vanitha Prabhakaran