# Background to the 32 bit architecture
The Windows 32 bit architecture separates memory into two areas, the user area and the system/kernel area. The system area is further broken down into areas with a number of different functions. Three of these are important to the scalability of Exchange servers, as the 32 bit architecture prevents us from using more memory than we have defined in the limits for these kernel resources.
For more details, see Table ‘A’ later in this document.
Due to the memory requirements of Exchange servers, the virtual memory is usually reconfigured from the 2 GB Kernel & 2 GB User memory ratio, to the 1 GB Kernel & 3 GB User memory ratio (using the /3GB boot.ini switch).
This change means that the kernel memory areas are reduced by 1GB to accommodate the larger user address space.
Table A: Kernel Memory and the /3GB Switch
Exchange and the use of kernel memory
Exchange servers must store security information in the form of tokens for each connection/logon that is made to the information store.
These tokens are stored in the Kernel memory space called the "Paged Pool". The Paged Pool is an area of virtual memory which can be paged to disk should physical memory resources become scarce.
Whilst tokens are not the only important objects stored in the Paged Pool memory, they are the most numerous. Every application that connects to the information store must logon, and in doing so security information is processed by the store and a Token is kept for the connection in the Kernel memory. In fact, each logon must store two Tokens due to the restriction of not being able to share security information between threads.
As the system allocates memory for Tokens, it does so in 8 byte increments, up to a Token allocation of 4 KB, after which the allocation size jumps by 4 KB. This means that once the token size goes over 4 KB (about 74 groups, at 55 bytes per group in the Token), the allocations will jump to 8 KB per Token due to memory management and alignment logic.
This tag is used by Windows to cache security information for every user session opened against the server. For example, a token (with memory allocated using the TOKE paged pool tag) is created for every user session generated by an e-mail client such as Microsoft Office Outlook. Depending on the client, multiple sessions may be opened for each e-mail client, causing multiple tokens to be cached on the server for each client. The paged pool memory footprint of each token is generally based on the number of security groups to which the user belongs. The more security groups a user is a part of, the more paged pool memory will be consumed by the tokens associated with that user's sessions.
In the MemSnap output listed in Table ‘B’, the average token size is approximately three KB:
Table B: Memory Snap output
The maximum recommended average Token size should not be allowed to go above 8 KB. To reduce the average token size the following actions must be performed:
--> Reduce the number of security groups in the organization.
--> Consolidate security groups and eliminate deep nesting.
Exchange Paged Pool Memory usage and topology considerations
Factors that influence the amount of Paged Pool memory consumed on an Exchange server are:
• Open files and objects
• Outlook Version and other MAPI clients (handhelds, scripts, Outlook plug-ins, Office Communicator)
• Desktop search engines (in use by clients connecting to the server)
• Outlook Calendar sharing
• Non-MAPI connections
• Total security group memberships
• Local Public information store
• Default public information store for other servers
• Offline Address books
• Free-Busy folders
If we look at the essence of the list above, we see that the two factors that influence paged pool usage are the number of connections and the token size that users have.
Some of the above are due to client software and configurations, others to software and activity locally on the server.
Open files and objects can be associated with busy messaging bridgeheads, where message queues are often likely to build.
When designing an Exchange infrastructure, server roles should be clearly defined, and multiple roles on a server should be avoided. Differing I/O and software requirements will mean that servers cannot be properly optimized if they share roles such as Domain Controller, Exchange Mailbox server and messaging bridgehead server.
Client Connections
Depending on the Outlook version, varying numbers of connections will be opened to the information store.
Additional connections may be kept open and some of the connections may even be generated and sometimes left open by client add-ons. These connections are the result of shared calendaring features, which retain additional connections to the server via background threads.
Scripts, handheld devices, search and index applications all increase the number of connections that clients are making to the store.
The SIDs for Security groups are added to the access token for users requesting access to resources on Windows servers and Public Folder configuration and "system folders", directly influencing the number of clients that will connect to an Exchange server.
Unfortunately, we do not have total control over the majority of connections made by clients to the Exchange server, other than by limiting the server topology, applications and features which the clients use when connecting. This means that the best way to control the paged pool usage on Exchange 2003 mailbox servers is to reduce the Token sizes where possible and reduce the number of connections.
Public Folders and associated considerations
Dedicated Public folder servers
Dedicated Public Folder servers can help isolate public store client connections, and remove the additional public store logons from mailbox servers, allowing them to support more mailboxes.
For large deployments, increasing the number of replicas of system folders will help spread the number of connections to any single instance of the system folders. Care should be taken to ensure that the servers to which these additional connections will be made (free-busy, Outlook security settings, offline address book) are capable of handling the additional load.
While dedicated public folder servers will be sufficient for most deployments, larger customers may need to consider dedicated system folder servers.
The location of replicas for the following system folders should be carefully controlled and planned:
--> Free Busy (there will be only 1 instance per admin group by default)
--> Offline Address Book (there will be only 1 instance per admin group by default)
--> EFORMS Registry (Org forms library only 1 instance per admin group by default)
In larger admin groups, if there are only single instances of the above system folders, they will draw significant numbers of connections to the servers holding their replicas.
In some circumstances, for large admin groups, it may be necessary to optimise the dedicated system folder servers, and remove the /3GB switch from the start-up options. This increases the Nonpaged Pool from 128 MB to 256 MB and the System PTEs from 24.000 to 300.000. Those memory areas are very important for System Folder servers like Free-Busy. If a Free-Busy server only hosts Free-Busy information the information store will be very small. In this case the Paged Pool memory space will not be a limitation and you can remove the /3GB switch.
By tuning the dedicated Free-Busy servers, you can reduce the number of replicas that you need to host Free-Busy information and improve the consistency of Free-Busy data.
Monitoring Kernel Memory utilization
If you plan to remove the /3GB switch from the boot.ini, to allow for more client connections, you must monitor the Exchange VM usage to be sure that you do not run into fragmentation issues.
You can expect a maximum Paged Pool of about 356 MB in this configuration and you should not allow the Paged Pool to grow above 320 MB in this configuration.
Monitoring Kernel Memory utilization
If you plan to remove the /3GB switch from the boot.ini, to allow for more client connections, you must monitor the Exchange VM usage to be sure that you do not run into fragmentation issues.
You can expect a maximum Paged Pool of about 356 MB in this configuration and you should not allow the Paged Pool to grow above 320 MB in this configuration.
- Sudhir
