Tuesday, January 31, 2006

How does OABGen know what domain controller to connect to?

Here is an article posted on EHLO by Dave Goldman.

In this article I will explain how the OAB Generation process discovers the domain controller that it will use for rebuilding an Offline Address List. Understanding this process can help you in troubleshooting.

 

As I have mentioned in other blog posts of mine, the System Attendant (MAD.exe) is the process that is responsible for invoking OABGen.dll (the OAB generation process). This happens one of two ways:

 

1. Via Schedule

 

To view the schedule for an Offline Address list you can do the following:

 

1. Open the ESM (Exchange System Manager)

2. Expand the Recipients Container.

3. Select the Offline Address List Container.

4. In the right hand pane highlight an Offline Address List.

5. Select Properties.

 

At the bottom of the Properties dialog box you will see an area that shows the Update Interval. This drop-down list box will control when the Offline Address List is generated. The list box will also show several predefined times:

 

·         Run daily as 2:00 am

·         Run daily as 3:00 am

·         Run daily as 4:00 am

·         Run daily as 5:00 am

·         Never Run

·         Use custom schedule

 

2. Via the ESM (Exchange System Manager)

 

1. Open the ESM.

2. Expand the Recipients Container.

3. Select the Offline Address List Container.

4. In the right hand pane right client an Offline Address List.

5. Select "Rebuild".

 

By selecting "Rebuild" you will force replication to occur. In doing so the System Attendant then becomes responsible for making sure that the selected Offline Address List is rebuilt.

 

But, the real question here is "how does the System Attendant find the domain controller??"

 

Well, in order for the OAB Generation process to connect to the Active Directory to make its NSPI queries, it must have a mechanism for doing so. This connection to the Active Directory will be facilitated by a MAPI profile.

 

What is a MAPI and what is a MAPI Profile?

 

MAPI stands for Messaging Application Programming Interface.

 

A MAPI Profile is a named list of message services and configuration data created by a user or service. A profile can contain any number of message services. Profiles will contain a session object and use service providers to do the work on the behalf of the user or system.

 

A MAPI Session is a period of fixed duration (logon time ---> logoff time) which the work is done. In the context of MAPI, work refers to a series of client requests to MAPI that are services by one or more service providers.

 

Once a scheduled rebuild or manual rebuild of an offline address list is performed, there are a few things that need to happen before OABGen.dll can even connect to the Active Directory:

 

1. The system attendant needs to force the Directory Service call to synchronize the Offline Address List.

 

2. After the Directory Service calls are made necessary security checks will be done to ensure that we have the proper credentials to make this call.

 

3. After the necessary security checks have been preformed we will make a call to the Active Directory to get the Configuration Domain Controllers DNS name. This is needed so we can obtain information like (Forest Name, Organization, Administrative Groups, Server Name, etc).

 

4. Once the Configuration Domain Controller's name is obtained a Session Object can be set up.

 

5. Now try to create one of the Directory Service objects, and read all of the information we need from the active directory (this is the data mentioned above).

 

6. Make a query to get the name of the offlineABServer.

 

7. Once the offlineABServer is found, get and use legacyExchangeDN to query that object in the active directory. (Example - "/dc=local/dc=OABGen/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=OABGen/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=W2K3-E2K3".

 

8. Next setup our tasks that are necessary for the OAB Generation (profile creation, obtain the public folder information store, etc).

 

9. Get the server name and add it to the MAPI Session that was previously created.

 

10. Logon to the Admin portion of the profile (System Attendant profile).

 

11. Obtain the system directory so we can find out where the MAPISVC.inf file resides.

 

12. Read in the service specific settings from the MAPISVC.inf file so the profile can be configured.

 

13. Check to see if we are a domain controller or not.

 

14. Get the mailbox name for this profile - "/o=OABGen/ou=First Administrative Group/cn=Configuration/cn=Servers/cn=W2K3-E2K3/cn=Microsoft System Attendant"

 

15. Now log profile on to our session object using the MAPILogonEx API. For more information on the MAPILogonEx API please refer to: http://msdn.microsoft.com/library/en-us/mapi/html/9764438a-0e36-4124-af1e-8296759ed216.asp

 

16. Open the Address Book provider by creating a new IAB interface object.

 

17. Associate this Address Book with this MAPI session.

 

18. Load all of our necessary providers here for the profile.

 

19. Open the profile sections and also initialize emsabp.dll.

 

20. Load the Address Book data.

 

21. Look up the AB Server that will be responsible for holding the Offline Address List.

 

Here is where the magic is:

 

22. Check to see if the Address Book server can be contacted and is online. If we find that we are online then we need to Logon to the Directory Service.

 

23. Check to see if this server is an Exchange 2000 server.

 

24. Check the registry to see if we are hard coding the profile by using the following registry key [HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\Exchange Provider]. Please see this Knowledge Base Article for more information:

 

ID Q314737 - XADM: Offline Address List Generation Does Not Work on Exchange 2000 Server SP1

http://support.microsoft.com/default.aspx?scid=kb;EN-US;314737

 

NOTE: Using this registry key is not a recommended or supported method, and should not be used unless directed by a Microsoft PSS Support Professional for troubleshooting purposes only!!

 

25. Open Address Book profile section in the MAPISVC.inf file.

 

26. Read from the MAPISVC.inf file and see if we can find the following MAPI property tag 0x6602001e = PR_PROFILE_HOME_SERVER. If this value is present in the DS Section of the MAPISVC.inf file then this is the server that we are going to use for our Active Directory connections via our MAPI profile.

 

27. If this value is not present we are an Exchange 2000 server, we will then use the RFR Interface (Referral Service). Using the Referral Service will allow us ask the Exchange Server to connect to the Active Directory and find us another suitable domain controller that we can connect to.

 

28. If the server is not an Exchange 2000 server or our Referral failed to get the profile home server address we need to rebind back to the Active Directory again.

 

29. Once we find our data save it to the profile.

 

30. Now that we have the necessary profile information to connect to the Active Directory we pass this data along to OABGen.dll.

 

31. When OABGen.dll is invoked to rebuild an address list, it will have use the domain controller that is will use for its Active Directory queries.

 

 

Monday, January 23, 2006

Microsoft Network Load Balancing: Frequently Asked Questions for Windows 2000 and Windows Server 2003

For those of us who implement Windows NLB on E2K3 Front End Servers, this is a good read.

NLB basics, NLB and other clustering technologies, NLB for scalability, NLB for high availability, NLB fundamentals, deployment issues, set up and configuration, management and operations, support for sessions, and application and protocol support.

Download At http://www.microsoft.com/downloads/details.aspx?FamilyID=3bc5c7d1-0c45-40a9-ac0a-84da23aa13b4&DisplayLang=en


Monday, January 16, 2006

Update for Intelligent Message Filter for Exchange Server 2003: 2006.01.05 released

Update for Intelligent Message Filter for Exchange Server 2003: 2006.01.05 (KB907747) has bee eleased.

Get it through Microsoft Update or WSUS

For information how to setup automatic updates, see:  http://support.microsoft.com/?kbid=907747

Microsoft IIS Diagnostics Toolkit

Overview

The IIS Diagnostics Toolkit is a combined release of popular tools used by today's IIS users. These tools include tools aimed at resolving problems related to Secure Socket Layer (SSL) issues, permission or security problems, gathering data for your SMTP server included with IIS, as well as the famous Log Parser utility used to sift through hundreds or thousands of log files very quickly.

The toolkit consolidates all the tools into a convienant download and is supplemented by updates every 90-days to ensure that users have the most current diagnostics tools at their fingertips.
 
Version: 1.0
Date Published: JAN/14/2006
Language: English
Download Size: 6.1 MB

Available also in amd64 and ia64 versions (see bottom of source page document)
 
The tools that are included in this MSI package are:
  • Auth Diagnostics
  • Debug Diagnostics
  • SMTP Diagnostics
  • SSL Diagnostics
  • Trace Diagnostics
  • WFetch 1.4
  • Log Parser 2.2
 

Friday, January 06, 2006

Exchange Does Not Always Use Local GCs

A good description from Jasper Kuria on why E2K3 does always use the local GC.

Until last week I was under the impression that Exchange always uses local GCs (Global Catalog servers) and only uses out-of-site directory servers (GCs or DCs) if no local ones are available. I found out otherwise from working on the issue described below.

 

One of our customers had a server outage and Outlook clients could no longer connect to the Exchange server. In other words the server was hung and we had a critsit (critical customer situation case). When dealing with hangs, the first thing we normally ask customers to do is capture a few hang dumps a few minutes apart so that we can determine what the hung process is doing.

 

The customer dutifully sent us 3 hang dumps taken about 5 minutes apart. On opening the dumps, the first thing that stood out was that there were 6 threads with similar call stacks whose execution state remained completely unchanged in all three dumps. In just one second of execution a thread’s stack can change so drastically that it is almost unrecognizable as the same thread.  Put in context, these threads had stayed at the same point for millions of years--in processor time. They each had a stack that looked like this (the stack grows upwards from kernel32!BaseThreadStart and only the relevant parts are included):

 

00 3386dbb8 74fd1394 NTDLL!NtWaitForSingleObject+0xb

...

05 3386ddc8 77955bb2 WLDAP32!LdapWaitForResponseFromServer+0x533

06 3386de04 77955a86 WLDAP32!ldap_result_with_error+0x101

07 3386de34 77959f71 WLDAP32!ldap_search_ext_sW+0x84

08 3386de90 77958f41 WLDAP32!LdapDetermineServerVersion+0x56

09 3386e22c 7795ef47 WLDAP32!LdapBind+0x1e4

0a 3386e25c 7795eed0 WLDAP32!LdapNonUnicodeBind+0x84

0b 3386e274 62ebcfbc WLDAP32!ldap_bind_s+0x17

0c 3386e2cc 62ebcefd dsaccess!CLdapConnection::BindToHost+0x100

 

...

 

16 3386e834 61ee2422 tokenm!CSearchResults::DoDCSearches+0x7b

 

...

21 3386f850 77d5d899 store!EcDoConnectEx+0x4c

22 3386f8c8 77d9c912 rpcrt4!Invoke+0x30

31 3386ffb4 7c57b388 rpcrt4!ThreadStartRoutine+0x18

32 3386ffec 00000000 KERNEL32!BaseThreadStart+0x52

 

From frame 05 (in bold), these threads were all waiting for a response from an LDAP server creating a convoy that was in turn blocking several other RPC threads. None of the threads was making progress and as a result the available RPC connections that Outlook clients rely on were maxed out. OWA clients worked fine while all this was going on because OWA does not use RPC.

 

So why wasn’t the LDAP server responding? First I dumped the LDAP connection object to determine what server these threads were querying. Interestingly, the server was named

 

                  aussydneygc1.au.company1.com

 

Per the naming convention, this was a GC in Australia yet the Exchange server was in the US. Hmmh, why were we going across the sea all the way to Australia when there were 16 healthy GCs in the US? The next thing I had them do was turn up diagnostic logging for DSACCESS (the Exchange component responsible for querying the AD) to find out what GCs were being discovered and in what order. DSACCESS uses an algorithm that discovers GCs in the local site first and then those in remote sites based on the cost of WAN (Wide Area Network) links and logs the list in the informational Event Id 2080 (viewable every 15 minutes when diagnostic logging for MSExchangeDSAccess à Topology is set to maximum). The table looked like this:

 

 

 

process INETINFO.EXE (PID=2380). DSAccess has discovered the following servers with the following characteristics:

 

 (Server name | Roles | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data)

 

In-site:

wsdc1.us.company1.com   CDG 7 7 1 0 1 1 7 1

wsdc2.us.company1.com   CDG 7 7 1 0 0 1 7 1

wsdc3.us.company1.com   CDG 7 7 1 0 0 1 7 1

wsdc4.us.company1.com   CDG 7 7 1 0 1 1 7 1

 

Out-of-site:

nydc5.us.company1.com   CDG 7 7 1 0 1 1 7 1

nydc7.us.company1.com   CDG 7 7 1 0 1 1 7 1

 

For more infomation on what these values mean see kb article 316300

 

Only the GCs/DCs in the US were listed (none of the ones in Australia were) and so DSACCESS could not have been the culprit.

 

I looked at the stuck threads a little more and determined that the users connected to the Exchange server via RPC had DNs (Distinguished Names) that looked like this:

 

"CN=John Doe,OU=Users,OU=Melbourne,OU=Company1 AUS,DC=au,DC=company1,DC=com"

 

"CN=David Boe,OU=Users,OU=Melbourne,OU=Company1 AUS,DC=au,DC=company1,DC=com"

 

 

"CN=Jane Zoe,OU=Users,OU=Sydney,OU=Company1 AUS,DC=au,DC=company1,DC=com"

 

For a moment I thought maybe these users in the Australian domain had mailboxes on the US Exchange server but the customer assured me that their mailboxes were properly located on Australian mailbox servers. The only other possibility was that these Australian users were logging on to the US server for public folder content.  Looking at the code and dumping the mdb (store) GUID confirmed this.

To recap the scenario, Australian mailbox users were connecting to a US Exchange server for public folder content. In order to determine what permissions the users had on the public folder content, the US Exchange server was querying a DC in Australia. The threads sending queries to this DC were stuck waiting for a response and they were in turn blocking other RPC threads, maxing out available RPC connections and adversely affecting Outlook clients on the US server. The question still remains, why wasn’t the US Exchange server querying the local US GCs listed above in the DSACCESS table?

It turns out that Exchange does not always use the local GCs. For certain specific security related user attributes like tokenGroups and tokengroupsGlobalandUniversal (used to determine what security groups a user is a member of and therefore what permissions s/he has to secure resources such as public folders). Exchange MUST query a DC that is authoritative for the user’s home domain, which will likely be an out-of-site DC—in this case it happened to be a DC in Australia. This behavior was introduced around the Exchange 2000 SP2 time frame to address an issue where users from remote domains (sibling or parent) were denied access to public folders even when the security groups they were in should have allowed them access. Pre-SP2 we had made the false assumption (in the product) that a local GC can service ALL queries that Exchange issues. A local GC can (and should) service MOST queries in a well designed multi-site AD environment.

Now back to Company1’s situation. At the time of the outage the WAN link to Australia was, in the customer’s words, "having some serious issues", which is why the LDAP responses were severely delayed. The connection was really spotty but it wasn’t down, which is probably why the connections didn’t time out. "How could this have been avoided?" they wanted to know. One simple way would be using dedicated public folder servers. If the US Exchange server were only a public folder server the worst that could happen is the Australian users wouldn’t have had access to the public folder content stored on the server (and perhaps individual Australian Outlook Clients would display the infamous "retrieving information from the server" RPC dialog). This is a less serious problem any day than lots of users (US users in this case) having no access to email. Another possible way to avoid the situation would be setting up replicas of the US Public Folder content on an Australian public folder server to avoid referrals over the WAN link. "But what if we have too little public folder content to justify a dedicated public folder server?" the customer asked. Fair question, but the honest answer is you risk running into this kind of problem again. Hopefully the WAN link going down is a relatively rare occurrence. If cross site public folder referral is also rare, the two rarities multiplied make for a low probability of the event occurring, but, like an earthquake, when it does occur...you get the point.

BTW we might have some good news on this in a few weeks. Stay tuned!

Thursday, January 05, 2006

Application Analyzer 2006 for Lotus Domino Beta

The Microsoft Application Analyzer 2006 for Lotus Notes/Domino is a tool that is used to help size and scope a Lotus Notes/Domino application migration project. The tool will analyze Lotus Domino applications, provide reports about application characteristics and recommend possible target solutions on the Microsoft Collaboration Platform.

Download At Source: 
http://www.microsoft.com/downloads/details.aspx?FamilyID=20e88f5d-b885-4ee0-8530-de12e96504b6&DisplayLang=en