Friday, December 30, 2005

Scripting Center help for MONAD

For everyone familiar with the scripting center that we have used for vbscripting and E2K3, there is now an extension to it for MSH scripting.  Enjoy...

Thursday, December 29, 2005

Large Security Tokens and Kernel Memory Exhaustion

This is a fantastic article posted by Mike Lee on EHLO.  It describes the exact issues with Kernel memory and Token bloat.  This is a MUST READ!



This is the third CXP flash about Windows 2003 kernel memory issues and Exchange 2003.


The first flash provided technical background about the demands Exchange makes on kernel resources.



The second flash discussed hardware configurations that can restrict the kernel memory available for applications.


This flash explains the effect of large user security tokens on Exchange's kernel memory usage.





Each client connection to a Windows server uses some paged pool kernel memory. The amount of kernel memory used per connection can vary widely. The size of the client's security token is the most important factor here.


Security tokens increase in size in proportion to the number of security groups to which a user belongs. This increase is generally linear, but there are sharp jumps in memory usage at certain thresholds.


There are an increasing number of Microsoft and third party clients and services that can connect to an Exchange server to provide expanded client, search and mobile functionality. Each of these clients may make multiple connections to the server, and each connection has token information associated with it. By reducing average token sizes, you can greatly increase the number of simultaneous connections that the server can manage.





  • Exchange administrators should monitor the amount of paged pool memory in use on Exchange servers during peak connection times, and reconfigure servers that are close to running out of pool memory. 
  • Exchange administrators should calculate the effects on pool memory of adding new mailboxes, clients or services to an Exchange server.


  • Exchange administrators should coordinate with Active Directory administrators to manage the number of security groups to which Exchange users are assigned. This is especially critical if the number of security group memberships is near a threshold where token size could suddenly jump.


Environmental changes such as the implementation of a new client or an increase in the number of security group memberships can suddenly increase kernel memory consumption to a critical level, causing the Exchange server to suddenly become slow or unstable. This can happen even though no changes have been made or mailboxes added on the Exchange server





What is a user token?


A security token is the bundle of information that identifies the user and the security groups to which the user belongs. Each time the user tries to connect to a secured resource, the token must be presented to the resource so that it can determine whether access should be granted or denied. For detailed information about security tokens please refer to the Access Tokens Technical Reference in the Windows Server 2003 Technical Reference at:


How big is a user token?


It depends. The most important factor is the number of security groups to which a user account belongs. Making a user a member of an additional security group can add up to 68 bytes to the token.


Various factors affect total token size and how many bytes are added per group. There is no simple mathematical formula for calculating token size based on the number of group memberships.


Nonetheless, the token for a user who belongs to 60 security groups is very likely to be less than 4K in size. The token for a user who belongs to 80 security groups is likely to be slightly more than 4K in size.


With regard to kernel memory consumption, there is a critical difference between a token that is "slightly less" than 4K and one that is "slightly more" than 4K.


As soon as the size of a token goes above 4K, even by a few bytes, it will take 8K of kernel memory to hold the token. If the token grows to slightly larger than 8K, its memory requirement will suddenly jump to 12K.


This increase may not seem that significant until you multiply it by thousands of clients--each of whom may make several connections to the server.


How much memory is taken up by user tokens on a typical server?


Consider an Exchange server where 1000 clients are logged on using Outlook 2003 in cached mode. Cached mode is a new Outlook feature that greatly improves the end user experience across slow or unreliable connections. Cached mode clients typically will not notice server disconnections lasting several minutes while online mode clients would experience errors in the same circumstance.


The typical Outlook 2003 cached mode client holds open 3 to 5 connections to the Exchange server. Older versions of Outlook (and Outlook 2003 in online mode) will make 1 or 2 connections to the server. Multiple connections allow Outlook to perform multiple server tasks in parallel. As a general rule, more connections per client means a better user experience. But, more connections per client also means more kernel memory consumption.


Each client connection to the Exchange store operates independently and needs a copy of the user token. There is also some RPC overhead for managing these connections, and busy clients may have more than one RPC connection to the server. Each RPC connection to the server also needs a copy of the user's token. Thus, each cached mode client has multiple copies of its token in memory on the server.


For the purposes of this example, assume that each client has a total of 5 connections and a token that is just under 4K in size. Each client then needs 20K of kernel memory.


This means that a thousand cached mode users will require about 20 megabytes of paged pool memory to support their connections. On a correctly tuned Exchange server with the /3GB switch set, there is a maximum of about 250 megabytes of paged pool memory available to all applications on the server.


NOTE: Please refer back to the first flash in this series for more information about the trade-offs involved in setting the /3GB switch. You can follow this link to read that flash on the Microsoft Exchange Team blog:


If additional security groups push the token size above 4K, then the amount of kernel memory used for tokens will suddenly be 40 megabytes instead of 20 megabytes. If each user starts using Microsoft Communicator and MSN Desktop Search,  there can be additional connections made to the Exchange server. If these additional clients add a single connection each, they increase paged pool demand by another 40%.


There may also be clients on your network who have Outlook add-ons that make additional connections, clients who connect to the server from multiple computers at once, and delegates who open multiple calendars or mailboxes simultaneously. Each of these puts additional pressure on paged pool memory by making additional connections.


What happens if I run out of paged pool memory?


The server will become slow or refuse additional requests and connections. Applications may fail suddenly. In extreme cases, the server can even "blue screen" and stop entirely.


If the paged pool shortage is transient, the server will likely recover. Applications can be somewhat resilient to temporary shortages of memory, but no application can run forever if critical resource requests are not satisfied. If the paged pool shortage lasts very long, it is likely to trigger cascading bottlenecks. In such a case, the server will probably have to be rebooted to make it functional again.


How much paged pool memory should be free in order for me to feel safe?


Under peak load, there should be approximately 50 megabytes of available paged pool. If you have less than 30 megabytes free, you should take immediate steps to reduce load on the server.


Paged pool is allocated statically at Windows boot time. The pool cannot be increased without reconfiguring and rebooting the server. The amount of paged pool memory available depends on a number of factors, including boot switches (such as /USERVA and /3GB), registry settings and physical RAM.


You can use a kernel debugger to view the size of initial paged pool and other kernel memory allocations. Setting up a traditional kernel debugging session can be a daunting task, typically requiring an extra computer, specialized cabling and a server reboot. Alternately, the LiveKD utility from Sysinternals can be used to start a kernel debugging session from the server console. LiveKD does not require you to reboot the server. For more information, please see this article in the Microsoft Knowledge Base:


The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003


IMPORTANT NOTE: Commands that can be used during a kernel debugging session can cause the system to become unstable or to stop. Microsoft recommends that you stop all Exchange services before initiating a kernel debugging session, and that you reboot the server after the session.


Without running a kernel debugger, you can still estimate how much paged pool is available on your server. A typical Exchange mailbox server with 1 gigabyte or more of RAM should have an initial paged pool allocation of slightly under 250 megabytes. (This assumes that the server has been configured with the /3GB switch and other recommended optimizations. Without the /3GB switch, initial paged pool on the same server would be about 350 megabytes.)


It is easy to check how much paged pool is currently in use. Windows Task Manager displays paged pool usage on the Performance page under Kernel Memory\Paged. You can also monitor paged pool usage over time with Windows System Monitor through the Memory\Pool Paged Bytes counter.


As a general rule then, paged pool usage in excess of 200 megabytes is cause for concern, and paged pool usage in excess of 220 megabytes requires immediate attention. If you are within these limits, and the server is still running out of paged pool, then the problem is likely that the initial paged pool allocation is insufficient. You can use a kernel debugger to verify whether this is the problem.


How can I find out how much of a server's pool usage is for user tokens?


You can find out how much kernel memory is being used for tokens at any given moment by using the Poolmon or Memsnap utilities. These utilities can be run without interrupting the server. They are available with the Support Tools for Windows 2000 and 2003.


Both utilities rely on the fact that each allocation of pool memory has a tag associated with it. The tag for tokens is called TOKE. Therefore, you can look through the output of either utility and find the TOKE line to see how much paged pool memory is in use at the moment for tokens.


NOTE: For Windows 2000, display of the pool tags is not enabled by default. You must enable tag display with the Gflags utility and then reboot the server before you can use Poolmon or Memsnap.


More information about monitoring and tuning kernel memory for Exchange is available here:


How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Memory Leaks


The "Ruling Out Memory-Bound Problems" section of the Troubleshooting Exchange Server 2003 Performance white paper


How to optimize memory usage in Exchange Server 2003


What can I do to reduce the size of user tokens?


There are three strategies you can follow:


  • Reduce the number of security groups to which each user belongs.


Nesting security groups will not help in doing this. The SID of each nested group is stored in the user token. In fact, if you are puzzled by what appears to be a discrepancy in the number of groups to which users belong and the size of their tokens, nested security groups may be the explanation. You can simplify group administration by following the best practice of not nesting groups beyond two levels.


If you are migrating user accounts from Windows NT 4 domains or between Active Directory domains, users may have sIDHistory entries for previous domains. If users belong to a large number of groups in that domain, this can greatly increase the token size. Completing the migration and decommissioning the previous domains may therefore reduce token size.


  • Host Exchange servers in a different domain than the users who connect to them.


This can reduce the size of user tokens by stripping domain local groups for the user account domain from the token presented to the Exchange server. This works because domain local groups from one domain are not kept in the token generated on a server in a different domain.


  • Where possible, convert security groups to distribution groups.


Token size is increased by membership in security groups, not distribution groups. Users can belong to thousands of distribution groups with no effect on token size. If a group is not actually being used to deny or grant access to resources, it should be a distribution group, not a security group.


What else can I do to reduce the total TOKE size on my server?


Once you have reduced the typical token size to the practical minimum, the next step is to manage the number of simultaneous connections made to the server. As mentioned above, each client may make multiple connections to the server, and different clients make different numbers of connections based on a wide variety of factors. You may not even have a full list of all the clients that connect to your server.


Users may install Outlook add-ons that make additional connections. Developers may run applications that make large numbers of connections or that don't shut down connections when they are done. Therefore, the first thing to do is to analyze what kind of clients connect to your server.


Exchange System Manager (ESM) will help you do this analysis. Each database displayed in ESM has a Logons page that lists the number of connections per logon, along with much other useful information.


You can get even more detailed information, including the name of each client process, by running the Exchange Server User Monitor (EXMon), which can be downloaded here:


After you have inventoried the connections being made, you are ready to consider the following actions to reduce the number of connections:


  • Restrict unauthorized clients and applications. 
  • Remove the Public Information Store from the server and direct clients to public folders on a different server. This eliminates public folder connections made by clients.


  • Remove specific public folders that account for large numbers of client connections. Good candidates here are the Schedule+ Free/Busy folder and the Offline Address Book. Clients must make additional connections to these folders when scheduling appointments or downloading the address book.


  • Add replicas of heavily accessed public folders to distribute the number of clients who connect to them between multiple servers.


  • Install dedicated public folder servers to eliminate all public folder connections from mailbox servers.


  • Distribute heavy connection users evenly across multiple servers. Heavy connection users are likely to be those with multiple computers or devices and field and mobile users.


  • Distribute users with large security tokens across multiple servers.


What is Microsoft doing to help with this problem?


Microsoft has made several tools and scripts available to help administrators manage and monitor kernel memory usage. A new sample script will soon be available that can scan Active Directory user objects and report how many groups each user account belongs to. When published, this script will be available in the following Microsoft Knowledge Base article. We will also publish the script on this blog site within next few days.


By knowing the number of groups each user belongs to, you can estimate how large user tokens are likely to be, and judge how close you are to a line where token size will suddenly jump by another 4K.


The configuration file for the Exchange Best Practices Analyzer (ExBPA) has been updated recently and now detects many problematic hardware configurations. You can download ExBPA here.


ExBPA is updated frequently to detect new issues and environments and to make recommendations to Exchange administrators for dealing with them. Each time you run ExBPA, it can check for new configurations and versions, and automatically update itself. Running ExBPA on a regular schedule gives you immediate and automatic access to the latest best practices for Exchange configuration.


The ultimate solution for these issues is to move to a 64-bit platform. The situation is similar to that which existed when the 16-bit DOS operating system was nearing the end of its life. Making even marginal improvements in memory management became important. While careful memory optimization on the 32-bit platform can improve scalability significantly, it cannot entirely overcome the fundamental problem of the 32-bit limit.


The next version of Exchange will be 64-bit. Until that version is available, Microsoft will continue to provide guidance and optimizations for maximizing the scalability of Exchange running in a 32-bit memory space.


Tuesday, December 27, 2005

Introducing the Microsoft Exchange Server Profile Analyzer

Posted on ...

Included in this month’s Exchange Tools Web release (direct link) is a new analysis tool called the Microsoft Exchange Server Profile Analyzer Tool (or EPA). We created this tool to help server administrators understand their user profile to help with the process of server sizing or capacity planning. A user profile describes how many actions an average user performs in an average day. When we refer to actions, we are talking about things that a user would do within an e-mail client application like send mail, delete mail, browse folder, etc. The EPA tool attempts to use whatever information is available in user mailboxes to generate an estimated profile, and depending on the characteristics of your users, the estimated profile may not be very accurate (for example, if messages are frequently deleted from the Sent Items folder, EPA will not generate accurate sent message statistics). For those of you who have worked with older versions of Exchange and Outlook, the EPA tool is building a profile in a similar way to the storstat.exe tool.

This package includes three separate tools:

  • EPA (available as EPAWin.exe and EPACmd.exe) is designed to generate a user profile based on the contents of user mailboxes.
  • EPAOWA is designed to generate a user profile by analyzing OWA log files
  • EPASummarizer is an assistant tool for EPA that can combine output files from EPA to summarize statistics across multiple data collections.  

This article will focus specifically on the EPA tool. We will explain how it gathers data, what data it gathers, and specific requirements for running the tool successfully.


The major steps that EPA takes to collect data are:


Step 1:Load topology from Active Directory. EPA reads Exchange configuration data such as organization, server, storage group, private mailbox store, and http virtual server from AD. This means that the account that runs EPA is required to have the ability to read Exchange configuration data from AD. In the GUI version of EPA (EPAWin.exe), you will see a tree view of the topology once this step is complete.


Step 2:Scan HTTP virtual server and virtual directory configuration for each server. For each http virtual server found on the server, EPA reads configuration information from the IIS metabase for two attributes: running state and SSL accessibility. For each virtual server, EPA maintains a list of mailbox virtual directory access links. The format of the link looks like this: http(s)://servername(:portnumber)/virtualdirectoryname/.

For a particular virtual server:

  • If EPA detects its state as not running, EPA will ignore it.
  • If EPA fails to read the running state, EPA assumes the virtual server is running.
  • If EPA reads that SSL is not enabled, for each mailbox virtual directory found on the virtual server, EPA adds a corresponding “http://” link to the list
  • If EPA reads that SSL is enabled, for each mailbox virtual directory found on the virtual server, EPA adds a corresponding “https://” link to the list
  • If EPA fails to read the SSL attribute, EPA adds two entries of this virtual server to the list. One assumes the SSL is enabled (https://), and the other does not (http://).

At the end, EPA sorts the list to put “https://” links on the top so that we will always attempt to use encrypted connections if they are available. When EPA moves on to Step 4 to collect data from each user mailbox, it will attempt to use each link in the list until it finds one that works. EPA will attempt to match the configured SMTP domain of a virtual server with each user’s SMTP domains to facilitate hosted Exchange scenarios.


You may notice that this step would require the account that runs EPA to have access to IIS metabase. If you don’t have access to the IIS metabase, EPA will make some assumptions about your configuration that should allow it to run under most circumstances, however you may see some error messages reported in the log if metabase access is not available. 


Step 3:For each server and mailbox store we found in the topology, EPA determines whether or not they will be included in the data collection stage based on configuration input,


Step 4:Data collection. Based on user’s configuration of the ServerThread and MailboxThreadPerServer attributes in the configuration file, EPA creates single/multiple working threads for data collection. If ServerThread is configured to 1 (which is the default setting), EPA will collect data sequentially on servers. Otherwise, it will create multiple threads for servers, and starts to collect data from multiple servers at the same time. Similar logic applies to MailboxThreadPerServer. The mailbox thread is the actual thread that is doing the data collection from a single mailbox (see the section below for the details of this function). After each mailbox is processed, the statistics generated from the collected data will be summarized to the parent mailbox store. The collection process for a mailbox store is done when all the mailboxes in the store are processed. The statistics for the server level are calculated when the collections for all the mailbox stores on the server are done. Similarly, the statistics at the organization level will be calculated when the collections on all the servers are finished.


How EPA collects data from single mailbox


EPA accesses items in user’s mailbox by using HTTP/Web Distributed Authoring and Versioning (WebDAV). This requires that the account that runs EPA has to have Full Mailbox Access. Note that EPA only supports Integrated Windows Authentication (NTLM) authentication at this time. If your server does not have Integrated Windows Authentication enabled for HTTP virtual servers, EPA will fail to collect data. EPA will also fail if HTTP virtual servers on back-end mailbox servers have OWA FBA (Forms-Based Authentication) enabled.


The primary WebDAV methodsthat EPA uses are SEARCH, PROPFIND, BPROPFIND and X-MS-ENUMATTS. The SEARCH method is used to get the hierarchy table and content table from a folder. To get the actual properties of a folder or of a message item, EPA uses the PROPFIND and BPROPFIND methods. X-MS-ENUMATTS is used to get the attachment table from a message item.  Table 1 shows the sets of properties that EPA reads for different types of store objects such as folder, message, attachment and appointment using WebDAV.  Table 2 shows a list of collectors EPA calculates based on the properties listed in Table 1.



Table 1: Properties EPA reads by using WebDAV

Property Name


Root Mailbox Folder























Folder Properties
















0x0FFF0102 (EntryID)

0x0e080003 (Size)

0x36010003 (FolderType)

0x663A000B (HasRules)

Message Properties






















0x0e060040 (DeliveryTime)

0x0e080003 (Size)

Attachment Properties

0x3703001f (Extension)

0x0e200003 (AttachmentSize)

0x3704001f (AttachmentName)

Appointment Properties



Note: Please refer to for a complete list of supported WebDAV properties and detailed descriptions of the data containined within the properties listed above.




Table 2: Exchange Server Profile Analyzer Data Collectors

Collector Name



The total size, in bytes, of the mailbox


Total number of rules defined in the mailbox


Total number of visible folders in the mailbox


Maximum number of messages in any one folder


Number of visible folders in the mailbox


Number of user created folders that are direct children of the root of the mailbox


Number of search folders in the current mailbox


The height of the folder tree


Average number of children each child of the folder tree root has.


Average height of each of the root folder’s children's subtrees.


Size in bytes of all the messages in a folder (takes list of folders to measure, e.g. "Inbox, Deleted Items, Sent Items")


Folder size statistics (across all folders in mailbox) - provides Avg, Min, Max.


Number of messages in the mailbox


Number of messages in a folder (takes a list of folders to measure, e.g. ("Inbox","Deleted Items","Sent Items")


Number of messages that are unread


Number of Deferred Action Messages in the mailbox


Number of messages where the subject prefix is "RE:" or equivalent subject prefix for the given culture


Number of messages where the subject prefix is "FW:" or equivalent subject prefix for the given culture


Number of messages containing at least 1 distribution list in the recipients table


Number of messages containing at least 1 attachment


Counts messages in a size range (takes a list of ranges, e.g. "2,10,100,1024" would provide counts of messages from 0-2,2-10,10-100,100-1024,1024-beyond)


Message size statistics across all messages (provides Avg,Min,Max)


Average number of messages received per day (provides Avg,Min,Max, and can restrict to the last N days)


Average number of rows in each sub table when the sent items folder is categorized by date (provides Avg,Min,Max and can restrict to last N days)


Average number of messages prefixed with "RE:" sent per day (provides Avg,Min,Max and can restrict to last N days)


Average number of messages prefixed with "FW:" sent per day (provides Avg,Min,Max and can restrict to last N days)


Number of messages in each body type requested (takes list of body types as input, e.g. "RTF,HTML,Other")


Average number of recipients of each message in the Sent Items folder


Average number of distribution list recipients of each message in the Sent Items folder


Attachment size statistics across all attachments (provides Avg,Min,Max)


Counts attachments in specified size ranges (takes a range list of inputs, e.g. "2,10,100,1024" provides counts for ranges 0-2,2-10,10-100,100-1024,1024-up)


Statistics on number of attachments per message (provides Avg,Min,Max)


Number of contacts in the mailbox


Number of contacts created per day (provides Avg,Min,Max, can be restricted to last N days)


Number of appointments in the calendar


Number of appointments created per day (provides Avg,Min,Max, can be restricted to last N days)


Number of meeting requests in the calendar.


Statistics on number of meeting requestes received per day (provides Avg,Min,Max, can be restricted to last N days)



We want to hear from you! Feel free to send your feedback on this tool directly to epafb AT microsoft DOT com and we will use your input to help make future versions of this tool even better. 


Frequently Asked Questions

Question: EPA reports CompletedWithException. Where can I find what exceptions occurred during the data collection process?
Answer:  EPA reports exception information exceptions in a log file. The default log file will be located at the user's application data path. For example, C:\Documents and Settings\<username>\Application

Question:  Error: Unable to connect to Active Directory. Please make sure that the user account has enough permission.
   Error: Unable to check the state of HTTP virtual server 1 on ServerName.
   Error: Unknown error (0x80005000) 
Answer:  The errors are reported because the account that runs EPA does not have permission to read IIS metabase. The error code may vary. EPA will continue to attempt to collect data from user mailboxes by making some assumptions about the configuration of your Exchange topology. See "Step 2" described above for details.

Question: Error: Unable to find an available URI for user ServerName\MDBName\Mailbox x.
Answer: The error is reported for two reasons normally. 
   1. No HTTP virtual server is running.
   2. For each HTTP virtual server on the exchange server, SMTP domain of the virtual server does not match with user's SMTP domains. 

Question: Error: User ServerName\MDBName\Mailbox x cannot access any of the following links:http://ServerName/Exchange/.
Answer: The error will come up in the following five cases. 
      1. The account that runs EPA does not have full mailbox access on the user's mailbox. This can be verified through Exchange System Manager. Please see the EPA documentation for details on how to configure and verify permissions.  If the account does have full mailbox access rights, then try using Internet Explorer to access the user's mailbox with OWA. If you are not able to access the mailbox with OWA, EPA will not be able to access it either. 
      2. The authentication methods configured on the virtual directory do not include Integrated Windows Authentication which is the only method EPA supports. 
      3. There are restrictions set up on the Exchange Server that restrict TCP/IP traffic in some way so that HTTP or HTTPS traffic from the machine you are running EPA on is blocked. You can work around this by running EPA on a machine that has the ability to connect with the back-end Exchange Server (such as an Exchange front-end server).
      4. When SSL is required on the server and the name on the SSL certificate does not match the name that EPA is using to access the server, EPA will fail due to SSL certificate validation errors.  We will have a fix for this in next Web Release. 
      5. When OWA Forms-Based Authentication is enabled on the HTTP virtual server, EPA will fail since the DAV requests submitted by EPA will get "440 Login Timeout" error. This is similar to;en-us;817379 . We plan to provide better feedback for this case in the next web release version.

If these troubleshooting suggestions don't solve your EPA problems, please contact for further investigation.