Tarpitting harvesters with Exchange 2003 on Windows 2003

New KB article (KB.842851) covers a Windows 2003 hotfix to tarpit anyone harvesting addresses from Exchange. You want this one if you’re doing Recipient Filtering based on directory lookups!

By default, Exchange 2003 will allow inbound SMTP mail to be addressed (“RCPT TO”) any alias within the defined relay domains. This means that even if you don’t have a user called “joe@domain.com”, if you do accept email for “@domain.com”, anyone sending to “joe” will just get a 2.1.5 SMTP code and presume the user does exist. Only after mail is processed at Exchange does an NDR get generated.

Those who send unsolicited commercial email send out lots of email addressed to mailbox aliases that don’t exist. Since Exchange will accept this email and then NDR it later, it puts additional load on your gateway servers to receive and process this mail.

One of the new features in Exchange 2003 was recipient filtering (see KB.823866). When you configure recipient filtering, there’s a checkbox you can check that says “Filter recipients who are not in the directory”. That’s great! That’ll solve the problem. Now instead of just blindly accepting the email with a 2.1.5 and later NDRing it, we can just look up in the directory right away and decline to accept the email (with “5.1.1 User Unknown”). Right?

Introducing a new problem: Directory Harvest Attacks. If all a spammer has to do to find out if an email address is valid or not is submit an “RCPT TO: <address to test>” command to your server, this becomes a great way to do email address harvesting. Exchange doesn’t honor VRFY, if you turn on this particular recipient filtering feature you get the same net effect! See this and this for more on DHAs.

The Windows 2003 hotfix in KB.842851 is an update for the SMTP stack used by Exchange. If you apply the hotfix and set a registry key, it will then start to “tarpit” any attempts to look up non-existent addresses. What this means is that if I send an email to UserThatExists@domain.com, the SMTP stream will proceed with no delay. But as soon as I try to “RCPT TO” UserThatDOESNOTExist@domain.com, the SMTP stack on the Exchange server will wait the specified (in the registry) number of seconds — say five — before responding. This obviously doesn’t mean that it’s no longer possible to harvest email addresses in this way, but it certainly makes it more “expensive” and therefore less attractive.

There’s a bit more detail in the hotfix KB, so if this interests you, have a look at it! Happy tarpitting!

 

Monad and E12 details

At this website, there is a good deal of information on the forthcoming Monad Shell (MSH) that is slated to ship with E12.  This will be a new WMI based command shell that is supposed to give us all the tools that ESM provides but in a command shell scriptable interface.  Here is a few of the key data points from this post.

• The integration of MSH and Exchange 12 is that the command line management of Exchange will be done via Cmdlets and that graphical management will be a layer on top of those cmdlets.
• The roots of Monad are WMIC - a command line interface to WMI. I invented that and was struck by the power and leverage that you could provide if you had common object reflection. In the summer of 2001 I was learning .NET and realized that we could do the same thing with .NET. That was the start of the project. No one understood what I was talking about so I had to develop an extensive demo to illustrate it. Once I did that, everyone started drooling and off we went.
• There are currently no plans to ship an SSH server. I am aware that other teams within Microsoft are investigating secure remote access, but I don't have details available.
• We are very influenced by the VMS (and AS400) environments. That said, we don't have the same set of utilities. Do define you own command, you write a .NET class derived from our base class and tag it with a NOUN and VERB. The properties of that class become PARAMETERS.
• Monad will be leveraging the WS-MGMT protocol for remote execution. WS-MGMT is an emerging industry standard that will provide a common protocol to manage servers in both the PRE-OS environment (e.g. talking to the base motherboard controller) and the OS-resident environment. It's pretty awesome stuff.
• We are working to ensure a path forward for VBScript/WSH users. There are 2 directions to that. One the one hand, Monad will provide VBSCRIPT-style scripting in that you’ll be able to access COM object and invoke methods and access their prosperities. On the opposite direction, we are looking at providing a COM interface to Monad so that VBScript users will have access to all of it's functionality. This is NOT a booked plan but should give you a feel for how we are thinking about the issue (so let's just keep this info between those of us with access to the internet ;) ) . We are working to get a booked plan and then make it public.

 

To download a preview of Monad technology go to http://beta.microsoft.com/content/bphome.asp. The GuestID is mshPDC.

MS IT: Leveraging dual SMTP Virtual server approach for Exchange 2003 gateway perimeter system

Here is a good basics article posted by Konstantin Ryvkin on how to configure the Exchange Server 2003 Front End servers for better control and reporting of mail traffic.

The vast majority of modern messaging environments have a challenge of establishing and maintaining reliable and secure Internet e-mail connectivity.  While the task of enabling Internet e-mail routing is relatively straight forward and usually well documented for most messaging platforms (most people view it as narrow as creating a DNS MX record for inbound e-mail and designing an equivalent of an SMTP connector for outbound e-mail), making a robust Internet e-mail system can be quite challenging.    In this blog, I wanted to share some of the issues and challenges that Microsoft IT faced in the Internet e-mail area as well as demonstrate some solutions that were implemented in Microsoft’s own messaging environment to make Internet e-mail flow.  This solution is based on Exchange 2003 technologies, so it may be relevant to other environments that have Exchange 2003 performing gateway functions to send and receive mail from the Internet.

 

Before we jump into the technical details, let’s look at the problem area.  In addition to the fact that a typical SMTP gateway system is a popular point of enforcing e-mail hygiene policies (such as antispam and antivirus) it has the fundamental task of bridging two different e-mail transports – a reliable, trusted, authenticated, rich and mostly homogeneous transport of the internal messaging system; and an unreliable, anonymous, generally non trusted, highly heterogeneous SMTP transport on the Internet or in the DMZ.  For example in case of Exchange 2003, the internal SMTP transport is authenticated with Windows Integrated Authentication, Send-As permissions are enforced, Linkstate and EXCH50 blobs are exchanged and hopefully viruses are spam are already being cleaned for internal messages.  The external SMTP transport does not exhibit such characteristics.

 

In essence the focus point of this blog can be pictured by the following diagram.

 

 

Most Exchange 2000/2003 environments face the following challenges when connecting an Exchange gateway to the Internet or to the DMZ SMTP infrastructures:

• Exchange SMTP gateway has to accept anonymous SMTP connections from the SMTP gateway in the DMZ or from the Internet.  Many of the environments do not want to expose Exchange SMTP authentication to the outside systems, to minimize the risk of password harvesting/guessing from the outside.
• Exchange SMTP gateway has to accept authenticated SMTP connections from the internal Exchange servers and relay e-mail received from them to the Internet.  It may also have to accept and transmit system information, such as link state and EXCH50 info, from the internal Exchange servers, which requires authenticated connection.  Allowing anonymous SMTP authentication to the internal SMTP systems also may be undesirable as it opens a possibility of unauthorized e-mail submission to that gateway. Anonymous mail submission, even in the internal environment, also opens a possibility of sender address spoofing.
• Exchange SMTP gateway administrator may want to enforce different actions (such as filtering, or archiving) to the SMTP e-mail that enters the environment versus e-mail that leaves the environment.  Inbound e-mail coming from the Internet is generally less trusted from the security perspective and often needs to be subject of additional security checks (such as spam filtering, content inspection).  Outgoing e-mail traffic is more trusted and some of the filtering/security controls may be waived for performance reasons.  In the end, you don’t expect your internal users to spam the Internet!
• Environments want quick and easy metrics and reporting on the amount of e-mail that they send to the Internet versus the amount of e-mail that they receive from the Internet.  Those metrics, for example, are often used for capacity planning and also to justify the investment into spam filtering technologies.

 

 

 

While Exchange 2003 SMTP stack is fairy functional and efficient in the gateway role, its default configuration makes it quite challenging to accomplish the above tasks.  Consider the following diagram that represents a typical default configuration of Exchange 2003 based SMTP gateway:

 

 

While perfectly functional from the mail routing perspective the above design has the following limitations/concerns:

• Both inbound and outbound mail go through the same virtual server making it difficult to enforce different policies, such as e-mail filtering, blocking or adding disclaimers.
• Authentication is exposed to the external environment making it possible for password guessing from external systems
• Relaying for authenticated users is allowed so knowledge of a single valid username/password allows external users to relay mail from the external hosts
• Inability to restrict the default SMTP virtual server by IP (for example so that only SMTP servers located in the DMZ can submit mail). Restricting by IP will prevent all internal Exchange servers from connecting.
• Anonymous SMTP is exposed to internal clients, that may quickly turn this gateway into mail server that internal users target with their SMTP applications to relay mail directly to the Internet bypassing established enterprise avenues for these functions
• There is no easy way to calculate metrics for inbound e-mail versus outbound e-mail (you most likely have to go through the tracking logs to get this data)

 

A slight modification of this design that Microsoft IT implemented internally for its Exchange 2003 based gateways makes all the above items possible.

 

 

Exchange 2000 and 2003 allows creating multiple SMTP Virtual servers on a single Exchange computer.  Obviously those SMTP Virtual servers need to be logically separated. One SMTP Virtual server is designated as Inbound SMTP Virtual Server and is handling e-mail traffic from the Internet and another SMTP Virtual Server is handling outbound traffic. The easiest way to create multiple SMTP Virtual Servers is to assign them to different IP addresses.  In the case of a dual homed computer those IP addresses are assigned to different NICs.  In the case of a server with a single NIC multiple IP addresses can be assigned to that single NIC.

 

When you create multiple SMTP virtual servers, they must be properly bound to the respective IP addresses in Exchange ESM.

 

Hosting the SMTP connector for the Internet on the appropriate SMTP virtual server, ensures that only that virtual server passes outbound mail going to the Internet.  Pointing MX records or smarthosting SMTP servers in the DMZ to the IP address of the second SMTP Virtual Server ensures that all e-mail traffic going through it is inbound to the organization.

An important note is binding SMTP Virtual Server to IP addresses affects only inbound SMTP connections.  The source IP address of the outbound SMTP connections from such gateway server will be determined by the IP layer of the sending host.  For example if the gateway has two IP addresses, 172.16.x.1 and 10.x.x.1 and wants to communicate with the remote host 10.x.x.2, then the source IP address of the SMTP connection will be 10.x.x.1, regardless which SMTP Virtual server established the connection.

 

Having such design in place you can apply different filtering and other security policies to SMTP Virtual Servers separately.  For example if you want to implement sender filtering of specific internal aliases (to mitigate the problem of them being spoofed from the Internet) you can enable such filtering on the Inbound SMTP Virtual Server only.  Thus, outbound e-mail flow will not be affected by such filtering.

 

If you have a spam filtering solution that runs on Exchange 2003 gateway platform (such as IMF or Brightmail) you can bind that solution to the Inbound SMTP Virtual server.  This way your outbound mail won’t be scanned for spam which could result in performance wins.

With the two SMTP virtual server approach you can control inbound vs. outbound e-mail traffic through this gateway separately.

 

As an added benefit of the above design, you get more convenient monitoring of inbound and outbound Internet mail flow going through this gateway.  Performance Monitor has separate instances of SMTP counters for each SMTP Virtual Server. The counter for instance  #1 represents the first SMTP Virtual Server (in this case Outbound mail) and instance #2 maps to Inbound Internet mail.

 

 

It should be noted that such configuration should be used for Exchange gateway servers that do not host production mailboxes.  When the server is rebooted the first SMTP Virtual Server to start will own the local store’s SendQ, hence for mail originating from the mailboxes hosted on the server itself there is no guarantee which SMTP Virtual Server will be used to process such messages. This may affect the validity of metrics for inbound vs. outbound mail.

 

Microsoft Preps for Exchange 'E12'

According to Microsoft Watch...

Next major Exchange update is still a year or two away. Before then, expect more Web tools and Service Pack 2.

After canceling a minor version of Exchange Server that was expected this year, Microsoft is now focused on making the next major version, code-named "E12," a deliverable in the 2006-2007 timeframe. E12 will include 64-bit support; new Web services programming interfaces; improved scripting and security; improved search functionality; and more e-mail/voice-mail/fax integration, according to company officials. The new release also will add better support for mobile devices, improvements in Office Web Access functionality; and better integrated meeting-scheduling capabilities, officials added. The E12 release will continue to rely on the Jet database store, as does the currently shipping Exchange Server 2003. Microsoft officials have expressed interest in unifying the SQL Server and Exchange Server stores over time. But at least for now, such a move is not in Exchange's foreseeable future, Microsoft officials acknowledged. Microsoft representatives declined to provide any firm dates for E12. Officials would not talk about their Beta 1 targets. But the final version of E12 could hit any time between 2006 and 2007, based on Microsoft's plan to release a new version of Exchange every three to four years. (The next version of Office, code-named "Office 12," is slated to hit beta 1 in late August 2005 and to go gold in the summer of 2006, according to sources claiming familiarity with Microsoft's internal shipping schedules. It is not clear whether E12 will be on a parallel schedule.) The Exchange team has been refining its roadmap over the past few months. Last summer, the team announced that it was planning a minor version of Exchange, known as Exchange Edge Services, for 2005. But in December, the Exchange team changed course, deciding to fold many of the features slated for Edge Services into either Exchange Server Service Pack 2 (due in the second half of 2005) or the next follow-on release (now known as "E12"). This week, Exchange team members made it plain that the Exchange team is not on the same schedule as the Windows Server team — which is working to release a new version of Windows Server every two years. The major Windows Server releases will ship every four years under this plan. The next major Windows Server release, Longhorn Server, is due out in 2007. (E12 will run on Longhorn Server, but will not require it or be optimized for it, Microsoft officials said this week.) Exchange Server customers tend to be a bit slower in their upgrade habits, according to Dave Thompson, the corporate vice president heading Microsoft's Exchange Server product group. Microsoft has listened to these customers, and is focused more on making their Exchange implementations smoother, rather than rushing out more new features, Thompson said. To achieve this goal, Microsoft is working on more Exchange Server 2003 add-ons, which the team will make available on the Web for download, in the first quarter of this year. Among these tools are ones for storage sizing assistance; SMTP configuration diagnosis; a Best Practices Analyzer for the Exchange pack for Microsoft Operations Manager 2005; and public-folder usage-analysis tools. In the second half of this year, Microsoft will roll out more ease-of-use and ease-of-management features, as part of Exchange Server Service Pack 2 (SP2). SP2 will include mobility enhancements and improved public-folder management facilities. It also will include Sender ID e-mail authentication integration to help fight phishing and spoofing, as well as updates to Microsoft's Intelligent Message Filtering (IMF) anti-spam technology.

Some more thoughts on disk IO and calculations….

Nicole Allen posted this on her blog back in November and I had read it back then but forgot to post it here.  Nicole has many great posts on her blog, so check it out, including her original post on Disk I/O.  
 
Measuring IO:  What to measure:
In my blog, I recommended that you measure the PhysicalDisk\Disk Transfers per second. Depending on your hardware situation, you may want to use LogicalDisk\Disk Transfers per second. If LogicalDisk counters aren’t enabled, you might be able to use the PhysicalDisk counters. The important thing, of course, is that you are measuring the amount of IO reads and writes per second to the database drives.  
 
I also recommend you measure IOs per second on the SMTP queue drives, log drives, and temp drive, because any of these drives may be a bottleneck. Most of the literature focuses on the database drives, because these tend to have the greatest rate of IO, but it’s important to remember the little guys too and make sure all the drives used by the Exchange server have enough throughput capacity to meet your company’s needs.
 
Measuring IO:  When to measure:
The question of when to measure keeps coming up. The important thing here is that the disks can support "maximum" sustained peak load. What do I mean by this? Practically speaking, sustained peak load is the load that is generated during the busiest time of the day, on the busiest day of the week. As I mentioned in the last blog, this is 9am to 11am on a Monday. When I monitor the servers, I calculate the average IOs per second during that busy 2-hour window. I am not interested in the maximum value that occurs during that time because I expect to see peaks and dips in IO rate - as long as these peaks are short in duration,  it is relatively safe to ignore them.  
 
So, if you have disk data from a many-hour measurement, I recommend looking at this in Perfmon. Perfmon allows you to select windows and find an average between two points. Find the largest sustained peak, and use the average IO rate from this peak in the calculations. 
 
Measuring IO: How many users is that again?
Determining how many users are active can be a tricky problem. It helps if you know a bit about the company’s email usage. For example, if you are lazy, and know that most of the users use mail, then you can use the total number of mailboxes for the number of users. Using the total number of mailboxes is what the Exchange 2003 performance and scalability guide recommends:
 
IOPS/mailbox = (average disk transfer/sec) ÷ (number of mailboxes).
   
This calculation works well in most cases. For example, people at Microsoft use mail a lot J, and during peak hours about 80% of the users are active, so this is a reasonable estimate for a company whose users have a passion for email. On the other hand, some companies don’t use email as frequently, or have users who work in shifts, so the number of concurrent active users might be much smaller.  In those cases, you may choose to use MSExchangeIS\Active User Count, or count the number of unique logons in ESM during peak hours.
 
In the last blog, I recommended that you use the MSExchangeIS\Active User Count counter. This is generally a good counter to use, but it does have a few gotchas I should mention. This counter gives the number of unique users that have logged on to the server and been active in the last 10 minutes. However, this number can be larger than the number of mailboxes on the server. This is primarily because of two things: if the server is a public folder server, it includes the users that are logged on to public folders. These can be users from other servers. Additionally, it includes users who are logged on to other user’s mailboxes (such as in the case of checking calendar details, if the users have shared their calendars).    This last reason usually has negligible impact - in most cases, it doesn’t account for many logons.
 
As an alternative, if you want to be more accurate about the number of users accessing a particular mdb, then you can look in the Exchange System Manager and count the number of unique logons for each mdb (drill down to administrative groups \<administrative group name> \ Servers \ <server name>\ Storage group\ <mailbox name>\ Logons).  Make sure you don’t count the same user twice.
 
When it comes down to how many disks you need, you only need to know the expected maximum disk throughput, which you can measure without knowing how many users are on the server. However, if you are planning to build a new server, and want to estimate how many IOPS to plan for, then it’s useful to know the IOs per user. As long as you are consistent in how you measure users, and know how many users you will have on the new  server, you should be able to estimate how much IO you will need - which is why we care about this in the first place. It might look like I’ve danced around the question "what is the best way to measure the number of users".  I suppose that’s because the answer depends on the situation. For simplicity, the method recommended in the Exchange 2003 performance and scalability guide is probably the best. I think MSExchangeIS\Active User Count is also good, especially if you have dedicated mailbox servers (with the public folder on another server), and it’s easy to measure quickly. And finally, if you are really detail-oriented, then you can measure unique logons via ESM.   
 
IO:  Where did the estimated maximum throughput numbers come from?

A couple people asked about the calculations, so I’ll put my little formula here:
 
Estimated maximum throughput =    D * F * T
 
D = disk speed. The maximum rate of IOPS measured by the disk manufacturer.
F = fudge factor. I used 0.8 (80% usage) to build in some overhead. This is necessary to plan for enough IO to handle occasional extremely high loads.
T = raid factor. This depends on the type of raid and the read/write ratio.
 
For no raid or raid 0, the raid factor is
 
T  = 1                                                               (no raid or raid 0)
 
For raid 10, the raid factor is:
 
T =   (R + W)/(R + 2W)                                   (Raid 10)
 
This ratio comes about because there are 2 disk IOs for every write.  Thus, the throughput is reduced by the ratio  (R+W)/(R+2W).
 
For raid 5, the raid factor is:
 
T = (R + W)/(R + 4W)
 
This ratio comes about because there are 4 disk IOs for every write in a Raid 5 configuration.
 
R and W are the number of reads and writes to the drives. You can calculate your own raid factor by measuring the number of reads and writes using the LogicalDisk\Disk Reads/sec and LogicalDisk\Disk Writes/sec. For example, if the number of reads to the database drives is 700, and the number of writes is 400, then the raid factor for Raid 5 would be
 
T = (700 + 400)/(700 + 4*400) = 1100/1900  =  0.48
 
Now I’ll walk through one calculation from my table. Let’s calculate the estimated maximum throughput per disk in a Raid 5 configuration for a R:W ratio of 3:1. Let’s use disks with a maximum raw throughput of 180 IOs per second, so D is 180. 
 
For a R:W ratio of 3:1, Raid 5, the raid factor is
 
T = (3 + 1)/(3+4*1) = 4/7 = 0.57
 
Thus,
 
Estimated maximum throughput =    D * F * T = 180 * 0.8 * 0.57 = 82.
 

Confused by different public folder tools?

Here is a great conparison between the most likely Public Folder tools available today posted by Bill Long.

With all the various public folder tools out there, it can be difficult to remember what tool works with what version of Exchange, and what the tool can be used for. Here's a summary.

pfadmin 1.3
Available at: ftp://ftp.microsoft.com/pss/tools/Exchange%20Support%20Tools/PFAdmin/
Used for: imports, or interactively
Works with: Exchange 5.5, 2000, 2003
Description: This is the original pfadmin from the Exchange 5.5 days. The most typical use is for running permissions imports that were generated with pfinfo 3.x, which are made up of SETACL command lines. You can also use it interactively to run SETACL and SETREPLICA commands. Pfadmin lets you view the current permissions and replicas using LISTACL and LISTREPLICAS respectively. However, these dumps can not be imported since they don't use SETACL or SETREPLICAS command format.

pfinfo 3.x
Available at: ftp://ftp.microsoft.com/pss/tools/Exchange%20Support%20Tools/PFInfo/55/
Used for: exporting permissions in pfadmin 1.3 command syntax, also reports certain properties
Works with: Exchange 5.5
Description: This is the original pfinfo from the Exchange 5.5 days. The most typical use is for running permissions exports, which are actually just a list of SETACL commands. These can be imported using pfadmin 1.3. Pfinfo also generates a report that includes various other properties of the folder, such as item count and folder size. The property export can not be imported. You can try running this against Exchange 2000 and 2003, but you will find that in some cases subfolders will appear as root folders in the export, because pfinfo gets the folder path from the directory object, which is not always populated in 2000 and 2003.

pfadminE2K
Available at: ftp://ftp.microsoft.com/pss/tools/Exchange%20Support%20Tools/PFAdmin/
Used for: imports
Works with: Exchange 2000 RTM only
Description: This is an updated pfadmin for Exchange 2000, but it is not widely used due to several shortcomings. First, its command and permissions syntax is completely different from pfadmin 1.3, meaning you can't import your old pfinfo exports with the new pfadminE2K. Second, with E2K Sp1 and later, most operations fail. It only works reliably with the RTM version of E2K.

pfinfoE2K
Available at: ftp://ftp.microsoft.com/pss/tools/Exchange%20Support%20Tools/PFInfo/2000/
Used for: exporting permissions in pfadminE2K command syntax, also reports certain properties
Works with: Exchange 2000 RTM only
Description: This is the updated pfinfo for Exchange 2000, which was also not widely used for similar reasons. The new permissions export format made it incompatible with any tool except pfadminE2K, and it only works reliably with E2K RTM.

OutlookFolders
Available at: ftp://ftp.microsoft.com/pss/tools/Exchange%20Support%20Tools/Outlook_Folders/
Used for: exports permissions in pfadmin 1.3 command syntax
Works with: Exchange 5.5, 2000, 2003
Description: This utility accesses Exchange via a MAPI profile, and thus it works with any version of Exchange. It generates permissions exports in pfadmin 1.3 SETACL command syntax.

PFDavAdmin
Available at: ftp://ftp.microsoft.com/pss/tools/Exchange%20Support%20Tools/PFDavAdmin/
Used for: exporting and importing permissions and replica lists in pfadmin 1.3 command syntax, version 2.3 and later exports properties
Works with: Exchange 2000, 2003
Description: PFDavAdmin accesses the information store through webDAV, thus the name, and was originally written to correct non-canonical DACLs on public folders. Its features eventually expanded to include imports and exports of permissions and replicas in pfadmin 1.3 command syntax. Version 2.3 and later can also export properties such as item count (PR_CONTENT_COUNT) and folder size (PR_MESSAGE_SIZE) into a tab-delimited file similar to the old pfinfo output (which also can not be imported, just like the old pfinfo output). In addition, it can be used to propagate single permissions and replicas down a tree of subfolders, instead of overwriting as ESM does. It has other features as well, which are discussed in the included doc.

pfmigrate
Available at: http://www.microsoft.com/exchange/downloads/2003/default.mspx, as part of the E2K3 deployment tools
Used for: adding and removing replicas for all folders on a server, site consolidation
Works with: Exchange 2000, 2003
Description: Pfmigrate is especially useful for Exchange 2003 Sp1 site consolidation, since it changes the legacy DN of a public folder, which no other utility does. Site consolidation is outside the scope of this article. However, pfmigrate can also be used to simply add replicas of all folders on one server to another server, and remove replicas of all folders from the original server.

exdeploy
Available at: http://www.microsoft.com/exchange/downloads/2003/default.mspx, as part of the E2K3 deployment tools
Used for: besides other things that exdeploy is used for, the /pubfoldcheck switch is of interest in regards to public folders as /pubfoldcheck runs a part of the DS/IS Consistency Adjuster that removes the unknown accounts from public folder ACLs. Those unknown accounts, if unhandled, could then get replicated to Exchange 200x servers as zombie users, thus causing problems. Exdeploy /pubfoldcheck can be scripted and run against multiple servers. NOTE: earlier version of exdeploy help stated that /pubfoldcheck also synchronized the public folder store and the Exchange directory (which could possibly cause public folders to be rehomed). This is however incorrect and it has been fixed in the new version of help file available with the tool at the above link.

BlogCast on using the Recovery Storage Group (RSG)

This is a new term for an old way of demonstrating something - BlogCasting.  Posting a video screen capture with audio on a topic.  In this instance, using the Recovery Storage Group presented by Ewan Dalton.  This link shows a second blogcast on the features of the Windows Smartphone.

Microsoft Exchange Server Best Practices Analyzer Tool update 1.6.7.1

In this ExBPA update, these changes are available: 

1.6.7.1 - Approximately 50 new rules added to detect the following conditions: Missing Exchange binary files, presence of recovery storage groups, availability check for the primary DNS server, Exchange process TCP port conflicts, and checks for over 25 non-default settings.

Is offline defragmentation considered regular Exchange maintenance?

Every so often clients ask me - "How often do you recommend we run the offline defrag?" Meaning - how often do we run it to have the server run OK?

The short answer would be - I do NOT recommend this to be run as any type of "regular" maintenance at all.

 

Looking at this closer - what really happens when you do an offline defrag?

 

1. You need to take your databases offline in order to run ESEUTIL on them. So - you are looking at downtime.
2. Defrag actually works by reading the original database, and copying used database pages into the brand new database file. When that is all done, we actually delete the original database file and rename the new one and copy it into original database file's place.
3. Not only are the used pages read, but they are renumbered/rechecksummed too. All secondary indexes in the database are discarded as well.

 

Seeing that we have a new database, we have new database signatures. That then means that transaction logs from that point on will have this new database attached to them. So - if you needed to do a restore a backup before the defrag and play in the transaction logs up to the current log - you would have problems, as transaction logs now talk about new databases (based on the signatures) and not the ones that you restored, right? Their signatures will not match and you will not be able to reply the data back in.

 

There is no real way around this. It is as if you had created a brand new database. It no longer has any relation to the old database, except for having the same stuff in it. You might as well have moved mailboxes to a new database.

 

Long story short - if you need to defrag - do it - but make SURE that you perform a good full (online) backup immediately after that!

 

Because of the above - offline defrag is definitely not something to do on regular basis. And it is typically not needed either.

 

So when DO I recommend defrag? There are a few situations:

 

1. If you have deleted a large amount of data out o the store and want to reclaim the hard drive space for whatever reason. This includes situations when databases reach the 16 GB limit on Standard versions of Exchange server.
2. If you had to run a hard repair of the database (eseutil /p - and that's another thing that we do NOT recommend unless this is a last possible thing to do). After running a repair, you should always offline defrag the database to get a new database file that has not been repaired. For more on what to do after the repair, please go here.
3. If you are experiencing a specific issue and have found a reference that says offline defrag will fix it.
4. If you are working with PSS and resolving the issue requires an offline defrag.
5. As a general rule, only defrag to reclaim space if you're going to reclaim more than 30% of the space. You can look for Event 1221 after nightly online defrag to get a conservative estimate of how much free space is in the database. For more info on Event 1221, please go here.

 

Some related reading:

 

256352 Online Defragmentation Does Not Reduce Size of .edb Files

http://support.microsoft.com/?id=256352

 

255035 XADM: How to Recover Hard Disk Space from Exchange Server Databases

http://support.microsoft.com/?id=255035

 

Best Practices for Exchange Database Management whitepaper:

http://www.microsoft.com/exchange/techinfo/administration/55/BestDBManage.asp

 

[You Had Me At EHLO...]

SMTP Session Tarpitting for Windows 2003 and Exchange

A common strategy for increasing the cost of would-be mail abuse uses a technique called tarpitting. Mail servers that tarpit wait a specified period of time before issuing SMTP responses to the client, thus increasing the time investment needed to successfully send a large amount of mail or a constant stream of (usually invalid) SMTP commands. To minimize the impact on the performance of well-meaning senders, servers can tarpit responses only for SMTP errors and allow authenticated clients to bypass the tarpit time.

Tarpitting is a useful countermeasure for:

• Dictionary harvest attacks (where an attacker is trying to compile a list of valid e-mail addresses from your organization)
• User account attacks (where an attacker repeatedly attempts to authenticate via username/password guessing)
• Spam scripts that send more invalid than valid e-mail recipients.

Most of these abuses depend on quick SMTP server responses to complete in an acceptable timeframe. SMTP servers that tarpit slow down the amount of work they can do in a given amount of time, thereby making the abuse less enticing or lucrative.

Until recently, there wasn’t a way to enable tarpitting behavior for Windows/Exchange. Now, you can.

Simply install the KB:842851 package and KB:885881 package. The only requirement is that you’re running Windows Server 2003 with Internet Information Services 6.0. If you’re running Microsoft Exchange, the package automatically integrates with it.

Then, create/set the following registry key:

            HKLM\System\CurrentControlSet\Services\SmtpSvc\Parameters\TarpitTime (DWORD)

The key value is the number of seconds you wish the server to tarpit error responses. You must stop/start the SMTP service for the change to take place.

When used with Microsoft Exchange Server 2003 features like recipient lookup, tarpitting increases the cost of invalid lookups that makes it harder to abuse the feature to launch a dictionary harvest attack.

[You Had Me At EHLO...]

Need ExBPA? Go to www.exbpa.com !

Courtesy of Paul Bowden, if you want to get information, updates or the ExBPA (Exchange Server Best Practices Analyzer) tool itself, you can go to www.exbpa.com, which will redirect you to the ExBPA page under the Exchange site:

 

http://www.microsoft.com/exchange/downloads/2003/exbpa/default.asp

That makes it a bit easier to find!

 

[You Had Me At EHLO...]

Upcoming February Webcasts

Here are the Microsoft Webcasts for Exchange Server 2003 in February 2005:

TechNet Webcast: How Microsoft IT Does Storage Design In Exchange Scale Up Deployments (Level 200)
Tuesday, February 01, 2005 8:00 AM (GMT-08:00) Pacific Time (US & Canada)
"This webcast will examine the storage design deployed within Microsoft to support the large-scale clusters utilized for the worldwide Exchange consolidation. Experts from Microsoft IT will outline best practices learned during the deployment and provide methodologies that will help you gain an understanding of what is required to deliver an optimized storage design."

https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032267609&EventCategory=4

TechNet Support WebCast: Performance baselines and troubleshooting analysis in Microsoft Exchange Server 2003
Wednesday, February 23, 2005: 10:00 AM Pacific time (Greenwich mean time - 8 hours)
"The objective of this presentation is to help customers understand why they need baselines during production deployments and how to analyze the data that is collected to identify potential areas of problems. We will discuss how to use the Performance Monitoring Tool and the relevant counters that are required to collect data and establish baselines. We will also review the data analysis and cover what counters to review to troubleshoot performance-related issues."

http://support.microsoft.com/default.aspx?kbid=891814

Microsoft Executive Circle Webcast: Enable Regulatory Compliance with Microsoft and EMC
Monday, February 21, 2005 11:30 AM (GMT-08:00) Pacific Time (US & Canada)
"Organizations worldwide are trying to navigate the minefield of e-mail retention, archiving, monitoring, and recovery to meet both business and regulatory requirements. Failing to protect e-mail has cost companies millions in litigation, regulatory fines, and lost production. All of these costs are avoidable by deploying an e-mail management solution that properly archives, stores, and monitors internal and external e-mail communications centrally and securely. Join EMC and Microsoft for a webcast to learn how to help minimize the risk of non-compliance, downtime, and lost data, while reducing costs and boosting the productivity of your entire Microsoft Exchange Server e-mail system."

https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032267119&EventCategory=4

[MS Exchange Blog]

Upcoming January Webcasts Reminder

Here are the Microsoft Webcasts for Exchange Server 2003 in January 2005:

TechNet Webcast: Performance Tuning for Exchange 2003 (Level 300)
Tuesday, January 18, 2005 8:00 AM (GMT-08:00) Pacific Time (US & Canada)
https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032266555&EventCategory=4

TechNet Webcast: Configuring Exchange and VPN Connectivity with ISA Server 2004 (Level 200)
Tuesday, January 18, 2005 9:00 AM (GMT-08:00) Pacific Time (US & Canada)
https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032265784&EventCategory=4

Microsoft Business Solutions Webcast: Smart Messaging with Microsoft Exchange 2003, Office System 2003 and Windows SharePoint Services (Level 200)
Thursday, January 20, 2005 11:00 AM (GMT-08:00) Pacific Time (US & Canada)
https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032266641&EventCategory=4

TechNet Webcast: How Microsoft IT Does Messaging Hygiene at Microsoft (Level 200)
Tuesday, January 25, 2005 9:00 AM (GMT-08:00) Pacific Time (US & Canada)
https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032266448&EventCategory=4

Introduction to Exchange Server 2003
Wednesday, January 26, 2005 9:00 AM (GMT-06:00) Central Time (US & Canada)
https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032267525&EventCategory=4

Small Business Webcast: Introduction to Exchange Server 2003 for a Small Office (Level 100)
Wednesday, January 26, 2005 11:00 AM (GMT-08:00) Pacific Time (US & Canada)
https://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?culture=en-US&EventID=1032266216&EventCategory=4

TechNet Support WebCast: Secure messaging in Microsoft Outlook
Thursday, January 27, 2005: 10:00 AM Pacific time (Greenwich mean time - 8 hours)
http://support.microsoft.com/default.aspx?kbid=891329

[MS Exchange Blog]

Microsoft rules out new Exchange data store?

SearchExchange.com released a story on Tuesday that says that rumors have been heard that MS has decided to not change the database engine in the next version of Exchange. 

"Microsoft has long talked about moving its Exchange messaging software to a new database technology, but it has now decided that the next major version of Exchange will not adopt a new data store.

Microsoft executives will publicly reveal a new roadmap for Exchange next week. In December, Microsoft said the technologies in Exchange Edge Services -- originally due out this year -- will now become part of the next version of Exchange. Microsoft has also said it now plans this year to release Exchange Server 2003 Service Pack 2, which will include the spam-fighting Sender ID technology."

Exchange Special Mailboxes Part 3 - SMTP mailbox

This post is the third post in a series. Special thanks to Scott Landry for his additions to this post.
You may also be interested in the earlier SystemMailbox and System Attendant mailbox posts.

 

SMTP (servername-{guid}) mailbox

What is it and what is it used for?

 

SMTP (servername-{guid}) mailbox is used by the mail transport of Exchange 200x as a temporary holding place for various messages as they pass through the system.

 

Inside the SMTP mailbox you can find the following folder structure:

 

 

The folders MTS-IN and MTS-OUT are fairly well documented in the Exchange 2003 SDK Message Transfer documentation. They are used by EDK connectors to facilitate the transfer of messages between the MTA and the Store.

 

So then, what about these TempTable folders? When a message is processed by Exchange transport, the message remains in one place while passing through the various phases and memory based queues. If the message originates from the SMTP protocol (another Exchange 200x server or the Internet), then as the message comes in, it is written via the NTFS store driver to the mailroot/queue folder while going through categorization & advanced queuing.  If the message originates from any where else -- MAPI client, OWA, or the MTA, then the message remains in the store during the same process, in the SMTP mailbox accessible to transport via the Exchange store driver using IFS.  No matter where the physical message resides during transport processing, the queue viewer in ESM will show messages from both places.

 

Where can it be found?

 

An SMTP mailbox will be created on each Mailbox Store. It’s stamped with the servername and the mailbox store GUID, so it should be fairly easy to identify. There is a mailbox object within the mailbox store, and a corresponding SMTP mailbox directory object in the Active Directory. The directory objects are found under the CN=Connections,CN=<orgname>,CN=Microsoft Exchange,CN=Services,CN=Configuration,CN=<domain> container.

 

When does it get created?

 

The SMTP mailbox is created in a mailbox store when the store is first created and mounted.

 

Should I delete it? Do I have to delete it before I can uninstall the Exchange server?

 

Nope, definitely should not delete it. If you’re getting the error “One or more users currently use this mailbox store. These users must be moved to a different mailbox store or be mail disabled before deleting this store. ID no: c1034a7f Exchange System Manager” when trying to remove the mailbox store or uninstall the server, this is not due to the SMTP Mailbox. Have a look at KB.279202 for more information on how to determine which mailbox is causing this behaviour.

 

Can it be moved and/or how do I recreate it?

 

It wouldn’t make much sense to move the SMTP mailbox, since it’s so tightly associated with the server name and GUID of a particular mailbox store.

 

If the SMTP mailbox is somehow deleted from a mailbox store, but the directory object remains, it’s as simple as dismounting and remounting the mailbox store. If the directory object is in place and is valid, the store will automatically recreate the SMTP mailbox object within the mailbox store. If it doesn’t you may need to have a look at KB.828938 for additional options.

 

If the SMTP mailbox directory object does not exist, the SMTP mailbox object in the mailbox store will become disconnected and will not be functional. You will see events 1022, 326, and 1194 logged in the application log. Generally if this happens, mail will just sit in the outbox. You’ll need to recreate the SMTP mailbox following the steps in KB.828938. Alternately (and a bit easier), create a new mailbox store and move users from the store without an SMTP mailbox over to the newly created mailbox store.

 

How can I log onto the SMTP mailbox?

 

This is possible, but you’re best to only do it under the advisement of PSS. There is a tool called Profinst which will create a MAPI profile to the SMTP mailbox. Note that there are several versions of this tool and if you run the wrong version (for instance, running the Exchange 5.5 version against an Exchange 2000 or 2003 server) it can cause problems. For Exchange 200x, you want the version dated 8–21–2001 (or newer), which should be about 28kb which can be found at: ftp://ftp.microsoft.com/PSS/Tools/Exchange%20Support%20Tools/Profinst/

 

What happens if I don’t have a proper SMTP mailbox?

 

If your SMTP mailbox is non functional, mail will not be able to pass through local delivery into the store or exit the store outbound for SMTP or EDK connectors. This will manifest as mail queuing up inbound to the store and as mail sitting (forever) in the outbox within Outlook. Note that this is not the only thing that can cause mail to queue or to sit in the outbox, and particularly if the mail eventually goes through it’s likely not the problem… but it’s definitely something to check if mailflow just stops. 

Update for Outlook 2003 Junk Email Filter (KB890854)!!!

Microsoft has released another Junk Email Filter update for Outlook 2003.  Sure wish they would do so for Exchange 2003 IMF...

 

Distribution Group Replication Issue and the ADC

I have been looking into an issue at a client that has had me baffled.  I was seeing this event:

Event Type: Error
Event Source: MSADC
Event Category: LDAP Operations
Event ID: 8270
Date:  1/7/2005
Time:  2:44:33 PM
User:  N/A
Computer: Server1
Description:
LDAP returned the error [41] Object Class Violation when importing the transaction
dn: cn=Object1,ou=Domain1,o=Company1
changetype: Add
member:cn=Test.Account,ou=Domain1,o=Company1
distinguishedname:CN=TestDG,ou=Domain1,o=Company1
createtimestamp:20050107200143.0Z
cn:TestDG
report-to-originator:TRUE
uid:TestDG
objectguid:B0C7F1EDFACAD54E82F1341FFB62EBD6
Replication-Signature:5F8ABEA8BE665A4A9CC6597A0D0563BA
ReplicatedObjectVersion:0
msExchADCGlobalNames:NT5:B0C7F1EDFACAD54E82F1341FFB62EBD6000000003666ADB1F9F4C401
: FOREST:69587472D51CF9469647EA9C5FE02C47000000003666ADB1F9F4C401
Replication-Sensitivity:20
proxy-Addresses:FAXmaker:TestDG@Company1.com
: SMTP:TestDG@Company1.com
: X400:c=US;a= ;p=company;o=City;s=TestDG;
-
  (Connection Agreement '2-way RCA' #1240)

I finally found an KB article that resolves this immediately.  Thank you Lonnie for the information.  It appears that the only thing I was missing was to place the Exchange 5.5 service account into the AD Backup Operators group.  It was already a member of the Domain Admins, but it appears that this doesn't give the SeRestorePrivilege permission.

 

Go figure.

 

Report: New Compliance Regulations Cause Boom In E-mail Archiving

InformationWeek posted an article yesterday saying that e-mail archiving applications sales increased from $33 Million in 2003, to $180 Million last year.  It is an interesting read...