Exchange 2007 SP2 RU1 – Raising diagnostic logging for “Message Access” causing calendar issues for users

Microsoft posted yesterday a new issue that administrators can cause with user calendaring just by increasing the diagnostic logging for “Message Access” to anything other than “Lowest”.  This seems to impact Exchange Server 2007 Service Pack 2 implementations up through Roll Up 1.  A KB article is not available yet, but appears to be in the works.  Based upon this Microsoft EHLO Blog article, the following information has been released on this issue. 

What the users may see

Symptoms before applying the pending update:

• Access to recurring appointments (which have attachments for the instances) is broken - Outlook in online mode receives an "Item cannot be opened" error.
• Sending an embedded message in cached mode results in the attachment being stripped.
• Availability is not shown for some users.

The following symptoms may persist, even after applying the update or manually setting the Message Access diagnostic level back to Lowest:

• Certain users show no availability information from Outlook or OWA scheduling assistant.  Also, event id 4009 for MSExchange Availability is logged on servers with the CAS role

Exception returned:

Microsoft.Exchange.Data.Storage.ObjectNotFoundException: Cannot open embedded message.

• Delegates viewing calendars receive the error:

Cannot read on instance of this recurring appointment. Close any open appointments and try again, or recreate the appointment

• Messages are sent to ActiveSync devices with the following text:

Microsoft Exchange was unable to send the following items to your mobile device. These items have not been deleted. You should be able to access them using either Outlook or Outlook Web Access.

• When accessing Calendar from OWA, the day, week or month viewing will fail with the error:

The item that you attempted to access no longer exists.

We have determined these symptoms are primarily due to calendar items affected between the time logging was increased and when the pending update or workaround is implemented. Recurring calendar items with no end date that have had an occurrence modified seem most susceptible.  A quick method to find these visually is to look for the circling arrows with a line through it.

Does this apply to you?

Before the release of the pending update, if any Exchange Server 2007 SP2 server with the Mailbox role has the following new event log level raised from Lowest, this applies to you.

MSExchangeIS\9000 Private\Message Access

How to check your Organization for the problem

You can determine if your MBX servers are at risk by looking in the following places:

1) The new GUI introduced in SP2 - in the Exchange Management Console under Server Configuration, Mailbox, select the server and choose Manage Diagnostic Logging Properties...

2) In the registry for each MBX server [Lowest = 0]

3) Run the following Exchange CMDlet to find all Exchange 2007 MBX servers and this specific diagnostic logging level for Message Access:

Get-MailboxServer | foreach {Get-EventLogLevel -id ($_.name + "\MSExchangeIS\9000 Private\Message Access")}

How to correct the problem

If any MBX server is found to have logging above the default before the pending update is applied, you should reset it to Lowest manually.  Note which MBX servers are configured with the non-default level and then run this CMDlet to ensure they are all set to "Lowest"

Then either remount the databases or restart the Information Store service.

Get-MailboxServer | foreach {Set-EventLogLevel -id ($_.name + "\MSExchangeIS\9000 Private\Message Access") -Level "Lowest"}

A sample PowerShell script is available here to track down calendar items contributing to the symptoms that persist after applying the workaround detailed above.  This script will identify the day containing problem appointments and can be run against a specific mailbox or all Exchange 2007 mailboxes.  The requirements for running the script are detailed in the script comments. The sample script uses the $true argument to enumerate all Exchange 2007 mailboxes and user42@contoso.com to initialize the Autodiscover portion of the Web Services object:

[PS] C:\Powershell\scripts> .\Find-BadCalendarItems.ps1 user42@contoso.com $true
Checking mailbox: user01@contoso.com
Checking mailbox: user02@contoso.com
...
Checking mailbox: user42@contoso.com
Checking mailbox: repro01@contoso.com
Failed: 11/30/2009 - 12/30/2009
Error: Mailbox logon failed., inner exception: Cannot open embedded message.

Day failed: 12/2/2009
Checking mailbox: repro02@contoso.com
Failed : 11/30/2009 - 12/30/2009
Error: Mailbox logon failed., inner exception: Cannot open embedded message.

Day failed: 12/23/2009
Checking mailbox: user43@contoso.com
Checking mailbox: lastuser@contoso.com
Problems found:
repro01@contoso.com: 12/2/2009
repro02@contoso.com: 12/23/2009
Done!

Now that 12/23/2009 has been identified as the problem date for user repro02@contoso.com, you can use Outlook to find any recurring calendar items with no end date that have had an occurrence modified on that day. Copy that occurrence [either to a temporary Calendar folder or even to a different time that day] then delete just that occurrence. Moving the copy back or manually recreating the instance will resolve the symptom for that user.