How to recover data from deleted mailboxes using Recovery Storage Group (RSG) in MS Exchange 2003

The Recovery Storage Group (RSG) is a new feature of Exchange Server 2003. In certain situations it removes the requirement of building up an additional recovery server in order to recover mailbox data (to recover public folder data, the recovery server is still required). However, when we restore the backup into RSG, we are unable to use the ExMerge utility to extract data from the mailboxes that have been deleted from the original mailbox store.

Overview of the Recovery Storage Group
Prior to Exchange Server 2003, if you wanted to recover some mailbox data from the backup, you needed to set up a separate Active Directory (AD) forest on a recovery server and restore the backup into this recovery computer. With the RSG feature, this additional recovery server is not required to recover data from a mailbox store in scenarios where both the following conditions are true:

• The logical information in AD about the storage group and mailboxes remains intact and same as before the recovery. That's to say, the mailbox you want to recover must not be deleted or purged from the system, or moved to another mailbox store.
• You want to recover data from a single mailbox, a single database, or simultaneously a group of databases within a single storage group. The database added into RSG must be from a server within the same Administrative Group; furthermore, after we add a database into RSG, we can then only add databases from the same Storage Group.

After you create a RSG and add databases to it, you can either restore online backup sets or copy offline database files to the RSG (please note that enough disk space is necessary for the recovered database in RSG). Then you can use the Exchange Server 2003 version of the Microsoft Exchange Mailbox Merge Wizard (Exmerge.exe) utility or the Recover Mailbox Data Wizard to extract data from the recovered databases in the RSG to the mailbox in the original regular storage group.

How a Recovery Storage Group links back to the original regular storage group
The RSG uses the following two AD attributes to link back to the original regular storage group:

• The msExchMailboxGUID attribute (see Figure A). When a mailbox is created, this attribute is generated as a unique value that distinguishes it from all other mailboxes. The value remains the same for the lifetime of the mailbox. The ExMerge utility uses this attribute to match the mailbox in the RSG to the one in the original regular storage group. When you delete a mailbox, the mailbox attributes will be removed from the AD user account; when you purge a mailbox that has been deleted, the mailbox (and its GUID) is removed from the database. In these two scenarios, ExMerge will fail to extract data from this mailbox in the RSG.

Figure A: The msExchMailboxGUID attribute

• The msExchOrigMDB attribute (see Figure B). This attribute is set on a database in the RSG. It specifies the distinguished name of the original database where the RSG was created, and it will verify if the mailbox you want to recover is still located in the original mailbox store. If the mailbox has been moved to another mailbox store, ExMerge will fail to extract data from the mailbox.


Figure 2: The msExchOrigMDB attribute

So, what is the problem?
As described above, you will encounter problems when you try to use ExMerge to extract data from mailboxes in the RSG under the following situations:
• The mailbox has been moved to another mailbox store
• The mailbox has been deleted or purged from the system

You may encounter the following error message when you try to use ExMerge to retrieve a list of mailboxes from a RSG:
An error was encountered with one or more users when retrieving the list of mailboxes homed in the selected databases on server 'ServerName'. Please refer to the log file, 'ExMerge.log', for more information.
And from the ExMerge log, you may see the following error:

[14:13:08] Error! Cannot identify the user with the msExchMailboxGuid C\03\96f\5D\B1\3BD\81\99\1F\91C2\5B7. The legacyExchangeDN is /O=ORGNIZATION/OU=SITE/CN=RECIPIENTS/CN=USER.

Problem Resolution
Depending on the scenario, you can use different methods to recover the data:

Scenario 1: The mailbox has been moved to another database
In this scenario, we can use the following two methods to recover the data from the mailbox.
Resolution: Move the mailbox back to the original database or modify the msExchOrigMDB attribute
The most efficient method in this scenario is to move the mailbox back to the original database where the RSG was created. Then ExMerge should work fine to retrieve the mailbox list and to extract the data from RSG.
If you are unable to move the mailbox back for any reason, you can modify the msExchOrigMDB attribute on the RSG database and point it to the database that you moved the mailbox to.

To do this, please follow the steps below:
1. Start the ADSI Edit utility. This is included in the Windows 2000 and 2003
Support Tools: click Start -> Run, type adsiedit.msc and press Enter.
2. Locate the mailbox store that you moved the mailbox to. To do this, expand the
following the containers:
Configuration Container [YourServerName.YourDomainName.YourTopLevelDomain]
CN=Configuration,DC=YourDomainName,DC=YourTopLevelDomain
CN=Services
CN=Microsoft Exchange
CN=YourOrganizationName
CN=Administrative Groups
CN=YourAdministrativeGroupName
CN=Servers
CN=YourServerName
CN=InformationStore
and then click CN=YourStorageGroup.
3. In the right pane, right-click the database object and click Properties.
4. Locate the distinguishedName attribute.
5. Right-click the value that is in the "Value(s)" box and click Copy.
6. Click Cancel.
7. Locate and click the RSG database object in the
CN=Configuration,DC=YourDomainName,DC=YourTopLevelDomain container.
8. In the right pane, right-click the RSG database object and click Properties.
9. Locate the msExchOrigMDB attribute.
10. Click Clear.
11. Right-click an empty area of the Edit Attributes box and click Paste.
12. Click Set and click OK.
13. Quit ADSI Edit.

For more information, refer to the following Microsoft TechNet article:
How to Change the msExchOrigMDB Attribute Using ADSI Edit
http://technet.microsoft.com/en-us/library/aa996434.aspx

Scenario 2: The mailbox has been deleted from the database, but not yet purged
Resolution: Reconnect the deleted mailbox to the original or a new user account
When a mailbox is deleted but not yet purged, it becomes an orphaned mailbox in the Exchange server database and is available in the Deleted Mailbox Dumpster (before the retention time expires). Under this situation, you can reconnect the deleted mailbox to the original user account and then recover the data.
If you cannot reconnect the mailbox to the original user (for example, you have created another mailbox and associated it with the original user account), you can just create a new user account (with no mailbox enabled) and then reconnect the orphaned mailbox to this newly-created user. This will work because the msExchOrigMDB attribute does not change on the mailbox even if we associate it with a new user account.

Scenario 3: The mailbox has been purged (permanently deleted) from the database
Resolution: Create a new storage group, or modify the msExchMailboxGuid attribute on a user to point to the mailbox we want to recover data from

If you are running Exchange Server 2003 Enterprise Edition, you can create a new regular storage group and copy the recovered database from RSG to this new storage group. You can then directly extract the mailbox data from the new storage group. For more detailed information see the following Microsoft Knowledge Base article:
You receive an error message when you use the Exmerge tool
http://support.microsoft.com/kb/919088

However, you cannot use this method on an Exchange Server 2003 Standard Edition because you are limited to one storage group in this edition. In this situation, you can directly modify the msExchMailboxGuid attribute on another mailbox to represent the one you want to recover (according to the principle of the msExchMailboxGuid attribute introduced above).

To do this, please follow the steps below:
1. The GUID shown in ExMerge log is in a different format than the actual
msExchMailboxGuid attribute. So first of all, you need to convert this GUID in
the ExMerge log into a format that contains 16 units with each unit having 2
hexadecimal numerals. Let's take C\03\96f\5D\B1\3BD\81\99\1F\91C2\5B7 as an
example:

i. For units like C\ and 03\, only 1 or 2 characters, leave it as is.
ii. For units like 96f\, 3 characters, split into 2 units: 96 and f.
iii.For units like 91C2\, 4 characters, split into 3 units: 91, C and 2.
iv. Following the rule described above, convert the original GUID into:
C 03 96 f 5D B1 3B D 81 99 1F 91 C 2 5B 7.
v. Then, for each single character, replace it with its ASCII code. So the GUID
is: 43 03 96 66 5D B1 3B 44 81 99 1F 91 43 32 5B 37. (You can visit
http://www.lookuptables.com to check the ASCII code.)

2. Start the AD Users and Computers snap-in, and create a new user with the same name and same organizational unit (OU) as referred to in the ExMerge log.
3. Start the ADSI Edit snap-in: click Start menu -> Run, type adsiedit.msc and
press Enter
4. Expand the Domain container, and expand the OU container which stores the new user (for example, CN=Users).
5. Highlight the newly-created user account in step 2 and open its Properties.
6. Locate the msExchMailboxGuid attribute.
7. Click Clear.
8. Copy the converted GUID: 43 03 96 66 5D B1 3B 44 81 99 1F 91 43 32 5B 37.
9. Right-click an empty area of the Edit Attributes box and click Paste to override with "43 03 96 66 5D B1 3B 44 81 99 1F 91 43 32 5B 37" (without quotation marks).
10. Click Set and click OK.
11. Quit ADSI Edit.
12. Run ExMerge again and connect to the RSG. This time we should see the mailbox
we want to recover. Follow the ExMerge wizard to extract the data.

Summary
In Exchange Server 2003, there is a new feature called Recovery Storage Group (RSG) which helps you to easily recover data (including messages, mailboxes or an entire database) without building up an extra recovery computer. However, due to the working mechanism of RSG, you cannot directly recover a deleted or purged mailbox. This KB provides several ways to get over this limitation and to recover the deleted mailbox through RSG.

Delayed Replication Advantage

# Advantage of putting DC’s on a delayed-replication schedule is that you can query the directory on the delayed DC to find information.

# E.g. the Distinguished Name (DN) which is required to restore the object, about the deleted item. You might need this functionality, for example, if a user account was deleted but nobody knows the user account's Organizational Unit (OU) path location. In a typical restore scenario, you would need to access a restored directory offline (not on the network) so that you could query for the object and gather the DN for use in the authoritative restore process.

# Delayed Replication of the AD environment allows DS to provide a higher degree of confidence that any lost objects, when detected will be recovered more quickly with less manpower.

- Sudhir

How to do Authoritative Restore?

Performing Authoritative Restores from the Delayed-Rep DC
The process for performing an authoritative restore using this delayed-rep DC is also pretty straightforward. Again, there is one minor complication:

1. Reboot the delayed-rep DC into Directory Services Restore Mode (DSRM). You’ll need the DSRM password to log in; you do know it, right? (If not, you can reboot the DC normally, and then use Ntdsutil to set the DSRM password.)

2. Once you log on, use Ntdsutil to mark the desired objects as authoritatively restored (more on that in a moment).

3. Reboot the server into normal operation.

4. Force outbound replication from the delayed-rep DC to the rest of the organization.

In Authoritative what’s complicated…?

=> Use Ntdsutil to mark the desired objects as authoritatively restored……..

# This’s the complicated part this time.
>> First, because it’s Ntdsutil, and not many people know/use Ntdsutil.
>> Second, because you must know the full DN of the object or objects you want to mark as authoritatively restored.

# How to find full DN of an Object?
>> Fortunately, you can use ADSI Edit to help you find the full DN.

What is Delayed Replication?

• The idea behind an Active Directory “delayed replication DC” (also referred to as a “slow DC” or a “lag DC”) is that organizations can more quickly recover portions of their Active Directory structure by performing an authoritative restore from this delayed replication DC instead of having to go back to tape.

• Keep in mind that many organizations use off-site storage of backup tapes, and recalling backup tapes from the off-site storage facility can take some time, as can the tape operations themselves.

• Let’s face it: when it comes to speed, tape isn’t exactly king of the hill.

• With a delayed replication DC (this would be a DC that only replicates on specific days of the week or after an extended period of time, like 72 hours), however, that process can be shortened drastically because recovering an OU, a user object, a group, or even AD-integrated DNS data can be done directly from the delayed replication DC.

• Once the delayed-rep DC is up and running, then you are ready to use it to perform authoritative restores of AD objects.

- Sudhir

Exchange 2003 Kernel memory usage

Exchange 2003 Kernel memory usage

# Background to the 32 bit architecture

The Windows 32 bit architecture separates memory into two areas, the user area and the system/kernel area. The system area is further broken down into areas with a number of different functions. Three of these are important to the scalability of Exchange servers, as the 32 bit architecture prevents us from using more memory than we have defined in the limits for these kernel resources.
For more details, see Table ‘A’ later in this document.
Due to the memory requirements of Exchange servers, the virtual memory is usually reconfigured from the 2 GB Kernel & 2 GB User memory ratio, to the 1 GB Kernel & 3 GB User memory ratio (using the /3GB boot.ini switch).
This change means that the kernel memory areas are reduced by 1GB to accommodate the larger user address space.



Table A: Kernel Memory and the /3GB Switch


Exchange and the use of kernel memory

Exchange servers must store security information in the form of tokens for each connection/logon that is made to the information store.
These tokens are stored in the Kernel memory space called the "Paged Pool". The Paged Pool is an area of virtual memory which can be paged to disk should physical memory resources become scarce.
Whilst tokens are not the only important objects stored in the Paged Pool memory, they are the most numerous. Every application that connects to the information store must logon, and in doing so security information is processed by the store and a Token is kept for the connection in the Kernel memory. In fact, each logon must store two Tokens due to the restriction of not being able to share security information between threads.
As the system allocates memory for Tokens, it does so in 8 byte increments, up to a Token allocation of 4 KB, after which the allocation size jumps by 4 KB. This means that once the token size goes over 4 KB (about 74 groups, at 55 bytes per group in the Token), the allocations will jump to 8 KB per Token due to memory management and alignment logic.
This tag is used by Windows to cache security information for every user session opened against the server. For example, a token (with memory allocated using the TOKE paged pool tag) is created for every user session generated by an e-mail client such as Microsoft Office Outlook. Depending on the client, multiple sessions may be opened for each e-mail client, causing multiple tokens to be cached on the server for each client. The paged pool memory footprint of each token is generally based on the number of security groups to which the user belongs. The more security groups a user is a part of, the more paged pool memory will be consumed by the tokens associated with that user's sessions.
In the MemSnap output listed in Table ‘B’, the average token size is approximately three KB:



Table B: Memory Snap output

The maximum recommended average Token size should not be allowed to go above 8 KB. To reduce the average token size the following actions must be performed:
--> Reduce the number of security groups in the organization.
--> Consolidate security groups and eliminate deep nesting.

Exchange Paged Pool Memory usage and topology considerations

Factors that influence the amount of Paged Pool memory consumed on an Exchange server are:
• Open files and objects
• Outlook Version and other MAPI clients (handhelds, scripts, Outlook plug-ins, Office Communicator)
• Desktop search engines (in use by clients connecting to the server)
• Outlook Calendar sharing
• Non-MAPI connections
• Total security group memberships
• Local Public information store
• Default public information store for other servers
• Offline Address books
• Free-Busy folders

If we look at the essence of the list above, we see that the two factors that influence paged pool usage are the number of connections and the token size that users have.
Some of the above are due to client software and configurations, others to software and activity locally on the server.

Bridgehead Servers

Open files and objects can be associated with busy messaging bridgeheads, where message queues are often likely to build.
When designing an Exchange infrastructure, server roles should be clearly defined, and multiple roles on a server should be avoided. Differing I/O and software requirements will mean that servers cannot be properly optimized if they share roles such as Domain Controller, Exchange Mailbox server and messaging bridgehead server.

Client Connections

Depending on the Outlook version, varying numbers of connections will be opened to the information store.
Additional connections may be kept open and some of the connections may even be generated and sometimes left open by client add-ons. These connections are the result of shared calendaring features, which retain additional connections to the server via background threads.
Scripts, handheld devices, search and index applications all increase the number of connections that clients are making to the store.
The SIDs for Security groups are added to the access token for users requesting access to resources on Windows servers and Public Folder configuration and "system folders", directly influencing the number of clients that will connect to an Exchange server.
Unfortunately, we do not have total control over the majority of connections made by clients to the Exchange server, other than by limiting the server topology, applications and features which the clients use when connecting. This means that the best way to control the paged pool usage on Exchange 2003 mailbox servers is to reduce the Token sizes where possible and reduce the number of connections.

Public Folders and associated considerations
Dedicated Public folder servers

Dedicated Public Folder servers can help isolate public store client connections, and remove the additional public store logons from mailbox servers, allowing them to support more mailboxes.

System Folders

For large deployments, increasing the number of replicas of system folders will help spread the number of connections to any single instance of the system folders. Care should be taken to ensure that the servers to which these additional connections will be made (free-busy, Outlook security settings, offline address book) are capable of handling the additional load.
While dedicated public folder servers will be sufficient for most deployments, larger customers may need to consider dedicated system folder servers.

The location of replicas for the following system folders should be carefully controlled and planned:

--> Free Busy (there will be only 1 instance per admin group by default)
--> Offline Address Book (there will be only 1 instance per admin group by default)
--> EFORMS Registry (Org forms library only 1 instance per admin group by default)
In larger admin groups, if there are only single instances of the above system folders, they will draw significant numbers of connections to the servers holding their replicas.
In some circumstances, for large admin groups, it may be necessary to optimise the dedicated system folder servers, and remove the /3GB switch from the start-up options. This increases the Nonpaged Pool from 128 MB to 256 MB and the System PTEs from 24.000 to 300.000. Those memory areas are very important for System Folder servers like Free-Busy. If a Free-Busy server only hosts Free-Busy information the information store will be very small. In this case the Paged Pool memory space will not be a limitation and you can remove the /3GB switch.
By tuning the dedicated Free-Busy servers, you can reduce the number of replicas that you need to host Free-Busy information and improve the consistency of Free-Busy data.
Monitoring Kernel Memory utilization
If you plan to remove the /3GB switch from the boot.ini, to allow for more client connections, you must monitor the Exchange VM usage to be sure that you do not run into fragmentation issues.
You can expect a maximum Paged Pool of about 356 MB in this configuration and you should not allow the Paged Pool to grow above 320 MB in this configuration.

Monitoring Kernel Memory utilization

If you plan to remove the /3GB switch from the boot.ini, to allow for more client connections, you must monitor the Exchange VM usage to be sure that you do not run into fragmentation issues.
You can expect a maximum Paged Pool of about 356 MB in this configuration and you should not allow the Paged Pool to grow above 320 MB in this configuration.

- Sudhir

Global Messaging

Exchange Server 200x is a complex messaging system with several Exchange core components and services which work together to provide an efficient email system. Exchange Server 200x highly depends on Microsoft Active Directory and a correctly functioning DNS system.
Exchange 200x services access information that is stored in Active Directory and write information to Active Directory. If this communication occurred directly between each service and Active Directory, Exchange 200x could overwhelm an Active Directory domain controller with communication requests.

-Sudhir