Deploying Exchange Server 2007 on VMwareInfrastructure: A VMware Internal Case Study

W H I T E P A P E R

2

VMware white paper

Table of Contents

executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

exchange Server 2003: The Legacy environment and Its Limitations . . . . . . . . . . . . . . . 3

Migration Drivers for Virtualized exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Planning and requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Design & Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

3

VMware white paper

executive SummaryVMware is an industry leading technology company. Email and instant messaging are mission-critical services that provide communications and messaging-based notifications for thousands of VMware employees. The Microsoft Exchange-based messaging systems at VMware support basic email and integrate with critical workflow tools, such as engineering notifications (build, release, bug tracking), shared document repositories (public folders) for internal user communities, and company-wide help desk functions. Email is a mission-critical medium for timely communication with key external communities including customers, partners, analysts and media contacts.

Before this project, the Exchange environment at VMware was based on a traditional, non-virtualized architecture, making it costly and difficult to maintain performance levels, enable high availability and resiliency against various failure modes, and manage downtime. Scaling the system and making optimal use of existing and new hardware technologies also proved challenging.

By virtualizing their Exchange Server 2007 deployment, VMware has gained the ability to:

1) Effectively eliminate planned downtime and maintenance/upgrade windows on every Exchange 2007 Server using the VMware HA, VMware DRS and VMware vMotion™ capabilities of VMware Infrastructure.

2) Execute immediate recovery from host, application and site failures through a combination of VMware HA and new clustering technologies (such as clustered continuous replication), eliminating single points of failure across various resource/host/application/site components.

3) Make the most efficient possible use of their virtual infrastructure by deploying virtual machines at greater densities with equal or better performance through optimal utilization of server, storage and network infrastructure.

4) Achieve exceptional scalability in a “building block” design, streamlining tasks such as adding user accounts, mailbox servers and other role servers while using robust virtual machine capabilities (design, placement, load balancing) to enable highly flexible Exchange services.

5) Accelerate Exchange life cycle and management functions with virtual machines and templates for rapid testing, provisioning, along with virtual snapshots and clones to help with rapid troubleshooting and problem reproduction.

The benefits realized from virtualized deployment of Exchange Server 2007 are a direct reflection of the VMware IT department’s “Virtualize First” policy that has resulted in more than 85 percent of VMware business applications running in virtual machines

today. The company’s ultimate goal is to virtualize 100 percent of its business applications on VMware Infrastructure to realize similar benefits.

The virtualized Exchange Server 2007 deployment at VMware supports more than 4 million messages per day, including 1.5 million external messages (spam included) and supports over 7,500 user mailboxes with a “Heavy Average User Load Profile.” The deployment allows good headroom for growth in a highly available and disaster resilient architecture.

The Exchange environment at VMware is deployed on 10 physical servers for all Exchange Server 2007 roles (approximately 40 virtual machines) in a multisite configuration. In comparison, the traditional non-virtualized deployment of Exchange Server 2003 required 14 physical servers at a single site for just the mailbox server roles. Enhanced end user experience, server-side throughput and availability of the Exchange environment are critical requirements that have been met and exceeded by the deployment.

This project has been an unqualified success for VMware, and reflects the experience of many VMware customers worldwide who are also running Exchange on VMware Infrastructure.

This white paper defines the high-level business requirements, design considerations and planning approach that were taken by the VMware IT staff, detailing the technical design and implementation process, and highlighting important operational policies for backups and availability.

exchange Server 2003: The Legacy environment and Its LimitationsThe legacy messaging environment at VMware consisted of Exchange Server 2003 Enterprise Edition in a traditional non-virtualized deployment. Two highly available failover clusters using the Microsoft Cluster Service (MSCS) were used to run all mailbox and public folder servers.

All Exchange Servers and Active Directory infrastructure servers were deployed in a single-site model. A total of 14 physical servers were required for mailbox roles supporting approximately 7,500 total mailboxes deployed in a single forest/domain/site AD topology. All other roles were deployed on virtual machines.

Details of the Exchange Server 2003 environment are as follows:

Network Infrastructure

• 1Gigabitpersecond(1Gbps)switchednetworkbetweenall clustered nodes, internal mail clients, and backup infrastructure

• 10Megabitpersecond(10Mbps)switchedconnectivityforcluster heartbeat communication via unmanaged network switch

4

VMware white paper

Active Directory Infrastructure

• TheExchange2003organizationwasdeployedinasingleforest, domain and site configuration (all Active Directory servers were virtualized)

Exchange Server 2003 Topology

• Front-EndServers

• Two(2)virtualizedExchange2003serversformultipleprotocols (IMAP, POP3 and HTTP)

• Hardwareloadbalancerprovidedevendistributionbetween clients and the front-end servers

• Back-EndServers

•Eightnode(6active/2passive)MicrosoftCluster

o Five (5) mailbox servers for all of North and South America users

o One (1) mailbox server serves all Asia Pacific/EMEA users

• Six-node(4active,2passive)MicrosoftClusterwithfour(4) mailbox servers for new hire employees of VMware

• Totalof7,500mailboxesonexistinginfrastructure

Client Systems and Protocols in use

• MAPI

• Outlook2000/2003/2007clients

• WebDAV/HTTPprotocol

• OutlookWebAccess(OWA)

• ActiveSync

• Entourage2004/2008(Microsofte-mailclientforMacOS)

• Evolution(E-MailclientforLinux)

• MiscellaneousIMAPclients

• Thunderbird(IMAPemailclientbyMozilla)

• Mail.app(emailclientforMacOSXandiPhone)

• Evolution(configuredasIMAPclient)

The legacy Exchange Server 2003 native-OS deployment posed significant limitations:

1. Scalability of the environment was a serious issue. The user population and email requirements at VMware had grown rapidly during the life of the system. The original Exchange Server 2003 environment consisted of 14 physical servers which could not adequately support 7,500 heavy user mailboxes.

• HeavyIMAPusage,sincecontent-conversiontookplaceon the mailbox servers.

• HeavyI/Oandlargemailboxeswereextremelycommon.

Adding more hardware was not cost-effective, due to expense for acquisition, management and operations, and a desire to reduce datacenter footprint.

Figure 1. Legacy exchange Server 2003 environment

5

VMware white paper

2. Traditional non-virtualized deployments precluded consolidation of the mailbox server instances to leverage larger and more powerful servers. Exchange Server 2003 and its32-bitarchitectureislimitedto4GBeffectivememory;it is unable to use modern large memory and cache effectively. This leads to stability issues that included memory fragmentation, small and inefficient use of allocated database cache and kernel exhaustion. In this situation, the only remedy is a reboot of the Exchange server and downtime for its users.

3. A single point of failure exists in Single Copy Clusters (SCC) deployed with SAN storage. Corruption of the information stores or SAN infrastructure inconsistencies such as improper zoning procedures, lose of connectivity etc are a significant vulnerability.

4. Physical server configurations are more difficult to provision and maintain. Provisioning new hardware to replace older Exchange Servers is extremely difficult. Reboots and system outages were required to complete simple server hardware upgrades.

5. Production backup schedules were difficult to meet. Backups took longer as users were added, and had a greater negative impact on the production systems during normal operation.

6.Storage,clusterorsitefailurewouldrequirerecoveryofthe data from tape, and an extended outage. Tape-based disaster recovery was adequate as a last resort, but imposed potentially significant down times that were not acceptable as an only option.

7. Early in the deployment of Exchange, mailbox quotas were not enforced. As the company’s growth accelerated, lack of mailbox quotas in the Exchange Server 2003 environment led to unpredictable server / storage utilization and mailbox growth.

8. Uneven distribution of mailbox loads required frequent mailbox movement between the mailbox servers to ensure proper load balancing and consistent end-user experience.

9. Availability levels were reduced overall because of the planned application and hardware upgrades, load balancing/scalability and stability issues.

Migration Drivers for Virtualized exchange The limitations inherent in the legacy Exchange 2003 deployment were significant reasons to take the opportunity to redesign and redeploy the messaging infrastructure in virtual machines, in addition to evaluating a transition to Exchange Server 2007. As it became obvious that Exchange Server 2007 in and of itself could alleviate some scalability issues, VMware took this opportunity to ensure that running Exchange 2007 on VMware Infrastructure would offer very significant additional

improvements in performance, scalability, availability, resiliency, life cycle management while reducing overall costs.

The main drivers toward the virtualized Exchange Server 2007 deployment at VMware are listed below:

VMware “Virtualize First” Policy

VMware follows a “Virtualize First” policy when evaluating application deployments and upgrades so as to realize important benefits enabled by VMware Infrastructure. These benefits include:

1) Dealing much more effectively with planned downtimes and maintenance/upgrades by using technologies such as VMware vMotion and VMware DRS.

2) Immediate recovery from host, application and site failures – combination of using VMware HA capabilities and traditional clustering technologies (MSCS) – to eliminate single points of failure across various resource/host/application/site levels.

3) Better use of hardware resources – VMware Infrastructure enables greater density of virtual machines that can optimally utilize various server/storage/network resource.

4) Highly scalable infrastructure – Adding users and mailbox servers/roles based on VMware Infrastructure is reduced to a simple building blocks approach based on robust virtual machine design/placement/load balancing/scaling models.

5) Expedite Exchange Management tasks and accelerate life cycle – virtual machines and templates provide rapid testing, provisioning;snapshotsandclonescanhelpwithveryrapidtroubleshooting and problem reproduction.

Whileevaluatingtheperformanceand64-bitarchitecturalenhancements in Exchange Server 2007 from an upgrade perspective, VMware concluded that combining the benefits of Exchange Server 2007 benefits with the benefits of virtualization benefits created a winning combination. This design improved agility for IT Operations by leveraging VMware Infrastructure capabilities such as vMotion and VMware HA.

Easier Testing and Prototyping

It was very important to accelerate the life cycle of design/test/stage/production and not be curtailed by the limitations of traditional non-virtualized deployments. As the initial design process was completed, testing the design under simulated load was essential to ensuring that Exchange 2007 would operate as expected. Testing in a virtual environment enabled very rapid changes, and effective testing on less hardware.

6

VMware white paper

Scalability and Server Containment

The demands of the messaging infrastructure at VMware and heavy mail usage user profile required more compute resources.WhileExchange2007isabletoexploitmodernlarge memory architectures more effectively, it still cannot be designed in accordance with Microsoft best practices in a non-virtualized deployment that utilizes the entire capability of a modern server. Using more, smaller servers would avoid wasting compute resource, but would increase expense for hardware, administration, datacenter space, power and cooling. Virtualization creates an opportunity to reduce server footprint and preserve the option to use more powerful servers later.

Application Stability

The inherent limitations of running Exchange Server 2003 in a traditional IT environment, along with the VMware “Heavy Average User Profile,” resulted in an unstable environment because of application downtime caused by memory fragmentation, the needs for planned hardware upgrades and related performance issues. The business requirement for a much more stable and inherently scalable environment was a major factor behind virtualized deployment.

Leveraging64-bithardwarecapabilitiesfullythroughExchangeServer 2007, and enabling better scale up on new and muchmorepowerful64-bithardwarewerealsoimportantconsiderations behind the virtualized deployment.

Enhanced Availability

Maintenance tasks are a fact of life for production servers. Unfortunately, they also require server downtime. As a worldwide company, there are no “off hours” at VMware. Someone will always notice a mail outage.

WithallExchangeserversinvirtualmachines,everyservercan be moved while running to allow hardware maintenance, upgrades or replacements without interruption to the service. VMware vMotion makes this a simple exercise. Additionally, VMware DRS will automatically move a virtual machine to an appropriate destination when a server becomes heavily loaded. These capabilities require virtualization, and can significantly reduce outages resulting from single-server issues.

Site Resiliency

VMware required a rapid response to any failure ranging from a single resource to loss of an entire site. The legacy environment relied on a single-site design with a tape-based recovery process. The combination of VMware HA and the application failover capabilities built into Exchange Server 2007 CCR can effectively eliminate single points of failure inherent in a non-virtualized implementation, and assure a rapid return of services in scenarios less dire than a true regional catastrophe.

Planning and requirementsKey Planning Considerations and Steps

VMware IT Operations teams began a proof of concept of Exchange 2007 on a fully-virtualized platform in October 2007. Detailed planning started in January 2008, migrations began March 2008 and the entire environment was fully virtualized and complete by June 30, 2008.

VMware IT staff enlisted assistance from various groups within VMware, including Performance Monitoring, Quality Assurance, Product Engineering and the VMware Business Solutions unit. Withthecooperationofthevariousteams,VMwareensuredthatthe planning phase resulted in a solid design framework that would meet or exceed clearly defined business requirements.

The approach VMware took in the planning phase was as follows:

1. Establish a baseline from the existing Exchange 2003 deployment to measure its capabilities and limitations across dimensions such as availability, performance, scalability, disaster recovery, and backups/restore.

2. Clearly define the overall workload for the Exchange 2007 deployment, and factor in key assumptions for scaling, growth and headroom.

3. Collect and rationalize commercial and operational requirements from across the company.

4. Translate and map the business requirements for scaling, availability, recoverability, backups, and other parameters into a set of architectural and design plans.

5. Size the infrastructure components, including servers, storage and networking.

6.Developatestenvironmentcomparabletotheproposedarchitecture,includingVMwareESXhosts,virtualmachines,storage sizing and network topology.

7. Validate the test environment using Exchange Server 2007 testingutilities(LoadGen,JetStress,ExBPA).Analyzeandremediate any observed anomalies, adjust the design as necessary and iterate through the tests.

8. Create a migration plan based on Business and Technical requirements using a pilot-to-beta-to-production approach.

9. Build the production environment based on the final validated test environment.

10. Transition existing Exchange 2003 mailboxes to virtualized Exchange Server 2007 production environment.

11. Monitor new production environment for anomalies, testing and correcting as necessary.

7

VMware white paper

Business Requirements

VMware places very high demands on its messaging systems andrequiresthattheybeavailable24x7x365.Asaresult,VMware had the following enhanced business requirements as it planned the migration over to the new Exchange Server 2007 messaging environment:

1. Multi-Site Resiliency – Provide a robust, reliable, resilient and mobile virtualized messaging infrastructure that is capable of withstanding site failure. Ensure that the infrastructure enables fast recovery from outages due to server or information store outages or corruption. Ensure the ability to maintain two (2) synchronized copies of the Exchange Server 2007 data.

2. High Availability – Elimination of any single point of failure in the architecture. Ensure that the system is capable of immediate failover of the messaging systems with minimal to zero IT interaction and without significant impact to connected clients.

3. Improved Performance – Reduce latency for connected clients. Ensure that significantly improved I/O performance in Exchange Server 2007 translates to a better end-user experience with faster Outlook response times and reduced RPC latency.

4. Scalability – Use VMware Infrastructure to increase hardware utilization efficiency, and to overcome the Exchange 2007 eight-core,32GBmaximumrecommendedconfiguration.

5. Zero-downtime upgrades – Use VMware vMotion along with Microsoft Clustering services to enable patching and upgrading with minimal impact to end users.

6.QuotaImplementation–Controlmailboxsizeandgrowthtoallow predictable utilization of storage resources and reduce administrative overhead associated with mailbox movement.

7. Flexible backup options - Provide flexible options around backups (with VMware Consolidated Backup, guest backups, array-based snapshots and passive node backups)

8. Support for multiple protocols – A significant percentage of the VMware mail users connect to the Exchange environment via IMAP, POP, and HTTP clients.

9. Reduce overall costs associated with operating and growing our user base/messaging needs, have a much more inherently scalable operational foundation.

10. Continue to support a very “Heavy Average Mail User” profile.

Design Considerations

Architectural Enhancements in Exchange Server 2007

ExchangeServer2007isdesignedtoexploit64-bitarchitectures.A brief list of advantages (more exhaustive information can be found at http://www.microsoft.com/exchange) include

• Abilitytouse64-bitmemoryaddresses–scalesbetterwith more memory - mailbox sizes and the number of user accounts per server increase.

• Reducedinput/output(I/O)requirementsenabledbythelargermemorycachesavailableon64-bitsystems–alsoleadsto better CPU scaling.

• TypicallyresultsinmuchbetterI/Operformanceandreduction in I/O requirements, which translates to more optimal use of storage.

Leveraging VMware Infrastructure Capabilities

The ability to scale Exchange mailbox servers on one physical server was a major driver for virtualization of the Exchange infrastructure.WithVMwareInfrastructure,theVMwareIT organization can utilize larger physical servers running VMwareESX3.5tohostmultiplemailboxservers,aswellasperipheral roles (hub transport, client access, etc). Stacking multiple virtual machines on powerful physical servers enables better hardware efficiency and utilization while helping to overcome the Microsoft recommended Exchange maximum ofeightprocessorcoresand32GBofRAM.Deploymentofnew Exchange 2007 virtual machines from preconfigured templates significantly reduced the time to provision. VMware HA and VMware DRS enable the required up time during both scheduled and unscheduled downtime for host patching and upgrades.

Very High Resiliency - Application Aware Clustering

The dependency of VMware on messaging as a mission-critical application, and resulting high availability requirements, drove the decision to continue use of Microsoft clustering technologies in conjunction with VMware HA.

VMware HA minimizes the duration of any outage by automatically restarting all virtual machines on another VMware ESXserverintheeventofserverhardwareoutage.Whileclustering can improve availability for mailbox servers, VMware HA protects all Exchange roles, improving overall availability when compared to a physical deployment. VMware HA minimizes any disruption to the messaging environment, and can be easily tested to ensure that the environment recovers as expected. VMware HA is simple to set up and protects every virtual machine without requiring complex clustering software. VMware HA was chosen to address the majority of cases where simple hardware failure in a non-clustered server can compromise overall operation.

8

VMware white paper

Exchange 2007 Cluster Continuous Replication (CCR) “is a high availability feature of Microsoft Exchange Server 2007 that combines the asynchronous log shipping and replay technology built into Exchange 2007 with the failover and management features provided by the Cluster service” (http://technet.microsoft.com/en-us/library/bb124521.aspx). Exchange CCR removed the architectural dependency on shared storage and eliminated exposure to a complete site outage as a single point of failure (a core requirement of the design). Exchange 2007 CCR on top of VMware HA allows VMware to have application-aware high availability, as well as host-level high availability. VMware HA automatically restarts virtualmachinesonotherVMwareESXhostsinavirtualclusterintheeventofanESXhostfailure.

Utilize Multicore Capabilities

Exchange deployments on physical servers are not able to effectively utilize the compute power now available in more powerfulquad-core(andmore)systems.VMwareESXhelpsto resolve this issue and enables significantly more scalability for the virtualized Exchange 2007 environment. VMware has documented (http://www.vmware.com/solutions/business-critical-apps/exchange/resources.html ) optimized deployment models and best practices based on a “building block” approach.

The building block VMware IT utilizes is based on four virtual CPUs. This is based on the 500 heavy-user-per-core benchmark which was achieved both in-house and externally. VMware decided on the use of blade technology to reduce the physical space and cooling requirements that were placed on the datacenter. The choice of more powerful blades allows the infrastructuretoscaleandreducesoverallTCO.Having16cores available in a single blade server enables deploying up to four active mailbox servers per blade and still leaves plenty of resources/headroom available to support passive mailbox servers and peripheral roles (hub transport, client access servers, etc.).

64GB of Memory per Blade Server

Using64GBofmemoryperbladeserverenablesmaintaininglowercostbyutilizingsmaller4GBmemorymodules.TheVMwareITmailboxserverbuildingblockutilizes12GBofmemory(2,000usersat5MBmailboxcacheand2GBfortheoperatingsystem);allotherperipheralrolesareallocated4GBofmemoryeach.Thesebuildingblockseasilysatisfytherequirements of the four active mailbox servers and can still accommodate the various peripheral roles that may run on anyonephysicalVMwareESXhostatonetime.

Storage Considerations

WhendesigningthestoragesolutionforthevirtualizedExchange 2007 environment, several factors were taken into consideration:

• Virtualdiskvs.rawdiskmapping–Internaltestingconfirmed that VMware could meet I/O requirements regardless of whether virtual disks or raw disk mappings were used. In some cases, virtual disks were slightly faster. Raw device mappings in physical compatibility mode was selected to accommodate array based clones or snapshots. This approach could also be facilitate “swinging” LUNS back to physical servers for troubleshooting. Note that with the release of VMware Consolidated Backup 1.5, which supports VSS, virtual disks can be used to deliver consistent and complete backups of virtual machines via snapshots.

• Performance–Disk/spindlecountisasignificantissuethat relates directly to the balance between designing for capacity versus throughput. Disk input/output rate (I/O per second, or IOPS) are a key design consideration for Exchange. IOPS demands may effectively prevent the use of fewer high-capacity drives, or require that those large drives be only partially utilized.

Forexample,300GBdrivesprovidehighcapacity,butreducetheavailableIOPSperGBofdata.Agroupof4x300GB15krpmdrivescandeliverapproximately500–600IOPSfor1,200GB.Choosing146GBdrivesinsteaddeliversthesameIOPSratefor584GB.IftheI/OrequirementperGBislow,thelowercostperGBoptionmight be appropriate. On the other hand, if the I/O requirementperGBishigh,systemperformancewillbeseriously degraded unless the smaller drives are selected or the larger drives are only partially used, wasting the extra capacity. This type of trade-off highlights the importance of working closely with the storage vendor of choice to determine optimal disk and I/O capabilities based on performance and space requirements.

• Disklayout&quotas–Thestorageconfigurationconformsto EMC and Microsoft best practices. The design utilizes RAID10 for all database and log LUNs, and ensures physical separation between the two to ensure recoverability and optimum performance. Each site has a single EMC CX3-80SANStoragearrayforstorageofthevirtualmachines and all local Exchange data. In the legacy VMware messaging environment, user mailbox quotas were not enforced. Several mailboxes had grown past 10GBinsize.Thisrequiredmoreadministrativeeffortto

9

VMware white paper

maintain manageable Exchange Server information stores by constantly moving mailboxes. The implementation of user mailbox quotas mitigates this additional administrative effort.Standardquotasof2GBand5GBareenforcedacrossall mailboxes. The majority of the user population maintains amailboxbelow2GB.Userswithspecialcircumstancesorbusiness justifications for an increased mailbox size can obtain 5GBofmailboxstoragewiththeappropriateapprovals.Nomailboxesabove5GBwillbesupported.

Design & ImplementationWiththetransitiontoExchangeServer2007,VMwarehasachieved a consolidation ratio of about four to one (4:1) with approximately 40 virtual machines running on 10 physical servers.Toplacethisintheproperperspective;10physicalservers run the entire virtualized Exchange Server 2007 organization including the domain controllers, anti-spam and perimeter e-mail routing servers. The legacy Exchange Server 2003 environment had 14 mailbox servers alone.

The Exchange 2007 environment is 100 percent virtualized on VMwareESXv3.5.ThevirtualmachinesrunWindows2003x64R2 SP2, Enterprise Edition.

Exchange Server High Availability

To meet the business requirements of no single point of failure, automated failover and multiple synchronized copies of the mailbox data and site resiliency, VMware combines VMware HA with a new Microsoft Exchange Server 2007 application-level high availability replication technology known as Cluster Continuous Replication (CCR). VMware HA will restart any virtual machine from a failed physical node on a running node in the HA/DRS cluster. CCR affords the ability to replicate the Exchange server data within the datacenter known as “in datacenter service” or to a remote site datacenter known as “site resiliency”.

One HA/DRS cluster is deployed in both of the VMware datacenters. The CCR design used at VMware populates each HA/DRS clusters with eleven CCR cluster-nodes. In the event ofanunscheduledESXhostoutage,theCCRclusterwillautomatically move the clustered mailbox role to the passive node in the second datacenter and continue to provide email services. Due to the VMware HA cluster, the failed active cluster-nodewillberestartedonanyremainingavailableESXhost,andCCR will initiate reverse replication to ensure that the cluster is in sync. The same methodology would be used in the case of a total site failure. To achieve complete autonomy between datacenters, a third site was chosen to house the file-share witnesses used by the CCR clusters to maintain quorum.

Utilizing the VMware Infrastructure Feature Set Optimally

To further improve upon the new design, the following capabilities within the VMware Infrastructure feature set were utilized:

VMware High Availability (HA) – Allows virtual machines to berestartedonothernodesofaVMwareESXServerclusterinthe event of hardware failure. VMware HA is utilized to ensure the virtual machines are powered on in the case of a server hardware failure.

Scenario:

• Hostserverfails;thiscausestheclustertofailovertotheremote site.

• VMwareHArestartsallExchangevirtualmachinesonanotherhost in the local cluster.

• Replicationcannowreversedirectiontoprimarysiteupdateto synchronize the Exchange. database.

• FailbackcanoccurmanuallywhentheExchangeAdministrator is available.

VMware Distributed Resource Scheduling (DRS) – VMware DRS continuously monitors utilization across resource pools and will intelligently allocate resources to virtual machines that require additional resources based on pre-defined policies. VMware DRS is used with full automation level for all Exchange server roles including clustered mailbox servers to allow for auto-placement based on resource requirements.

Scenario:

• ExchangeMailbox,CASorHUBserverorserversrequireadditional compute resources

• VMwareDRSscanstheclustertodeterminethehostthatcanbest satisfy the request for additional resources.

• Virtualmachineswillbemigratedtothehostintheclusterthat will satisfy the request using vMotion.

• ThisresultsinahighlybalancedandoptimizedExchangeconfiguration.

VMware vMotion – Allows virtual machines to move from one physical server to another physical server with no impact on operation or end user services. VMware vMotion is used to move virtual machines between physical servers within the VMware HA cluster for physical server maintenance and patching. All of the virtual machines in the environment (including active mailbox servers) can be migrated under load with vMotion.

10

VMware white paper

Scenario:

• AVMwareESXhostneedstobeupdatedviapatchesandadditional memory must be added to the host.

• AffectedESXhostcanbeplacedinMaintenancemode.

• VirtualmachinesmigratetootherESXhostsintheclusterfreeing up the affected host.

• ESXhostispatchedandadditionalmemoryisadded.

• ESXhostisrestartedandvirtualmachinesaremigratedbackto upgraded host.

Mail Routing and Client Access

VMware uses a centralized infrastructure and administration model. In this model, the majority of infrastructure services are served from two datacenters near the company’s main campus in Palo Alto, California. The two-datacenter design of the Exchange 2007 infrastructure and the nature of mail routing in Exchange 2007 led to an interesting dilemma - one or two Active Directory sites?

One Active Directory site would allow for simplicity, fewer domain controllers, fewer overall Exchange servers and fewer customizations. However, it was possible that mail between two mail users whose mailbox resided on the same mailbox database could potentially bounce between the Palo Alto and Santa Clara datacenters. Two Active Directory sites would allow for all mail local to Palo Alto or Santa Clara to stay within their respective datacenters usinglocalhubtransportservers.WechosetouseatwoActive Directory site configuration.

The two Active Directory design also meant changes to how client access worked. In this design, all connections toSantaClaramailboxesviaOWA,ActiveSync,andOutlookAnywhere are proxied to Santa Clara through Palo Alto. IMAP and POP connections are also proxied through Palo Alto, which required customization of the default IMAP and POP Exchange Server 2007 Environment

Figure 2. Client access and mail routing at VMware

11

VMware white paper

Exchange Server 2007 Environment

The new Exchange Server 2007 environment runs entirely in virtual machines on VMware Infrastructure 3. The virtual machines are deployed as 22-mailbox cluster nodes that make up 11 Cluster Continuous Replication (CCR) clusters. This houses 11 highly available Exchange Server 2007 clustered mailbox servers (CMS). The cluster nodes are geographically dispersed between the Palo Alto and Santa Clara datacenters. This design allows for site resiliency and eliminates single point of failure. Each site has dedicated hub transport, client access and domain controller/global catalog servers. To efficiently route traffic between the various transport servers, a second Active Directory site was created for Santa Clara. The addition of this site helps avoid mail from unnecessarily being sent across the metro link.

Storage Subsystem Architecture/Design

The storage configuration leverages our storage vendor’s best practices. The design utilizes RAID10 for all database and log LUNs, and ensures physical separation between the two to ensure recoverability and optimum performance. Each site has asingleEMCCX3-80SANStoragearrayforstorageofthevirtualmachines and all local Exchange data.

DatabaseLUNsareconfiguredusingfour(4)300GB/15kFCspindles. Each database LUN houses two (2) mailbox databases for a maximum of 350 users. The standard mailbox server building block uses seven (7) database LUNs configured as physical mode RDMs. The four spindle LUN allows us to satisfy our 1.15 per mailbox I/O requirement.

TransactionlogLUNSareconfiguredusingten(10)300GB/15kFC spindles. The single log LUN is configured as a physical mode RDM and is shared between the 14 storage groups that reside on the standard mailbox server building block.

System/boot partitions use virtual disks on VMware VMFS volumes configured for RAID5 at the array level.

Figure 3 – Virtual Machine Disk Layout

12

VMware white paper

Performing Backups in the Exchange Server 2007 Environment

Daily backups are performed using traditional Exchange streaming backups to virtual tape libraries (VTL). The use of VTLs allows VMware IT to keep 30 days worth of backup data online and quickly accessible. Due to the tape-free design, there is no risk of having tapes off-site when needed or having to locate all of the different tapes used for a particular save-set. This allows for the implementation of a less stringent backup schedule such as weekly full and daily incremental (or differential). Furthermore, recovery from alternate backup media would only be required in the event of multiple failures, one at the primary (active) data source and one at the alternate (passive) data source.

Figure 4. Backup environment

Figure 5. Backup Schedule

Sun Mon Tues Wed Thurs Fri SatSG1,8 Full incr incr incr incr incr incrSG2,9 incr Full incr incr incr incr incrSG3,10 incr incr Full incr incr incr incrSG4,11 incr incr incr Full incr incr incrSG5, 12 incr incr incr incr Full incr incrSG6, 13 incr incr incr incr incr Full incrSG7, 14 incr incr incr incr incr incr Full

13

VMware white paper

Hardware Configuration

Figure 6. hp c7000 Blade Chassis

Storage Area Network Configuration

Figure 7. SaN Configuration

14

VMware white paper

Figure 8. VMware eSX Servers/exchange infrastructure Virtual Machine

Figure 9. exchange Server 2007 Cluster and Site Configuration

15

VMware white paper

ConclusionsVMware has successfully deployed a fully virtualized Exchange Server 2007 environment on VMware Infrastructure. This deployment has met and exceeded the business and technical requirements that were defined and which have been outlined in this paper. The benefits of virtualization combined with the benefits of Exchange 2007 offer capabilities and cost savings that are not otherwise available.

By virtualizing its Exchange infrastructure, VMware has gained the ability to:

1) Easily scale up and continuing to grow the deployment as the company’s base of users grows.

2) Deal efficiently with discontinuous points of growth, such as acquisitions and rapid user group provisioning/data migrations.

3) Continually monitor and optimize the physical and virtual infrastructure to continue to drive down costs associated with space, power/cooling and enabling the transition to “green IT.”

4) Evaluate and leverage advancements in hardware technologies with minimal/zero disruption on the production Exchange environment.

This project has been a complete success, and reflects the consistently positive experience that VMware customers report after deploying Exchange 2007 on VMware Infrastructure.

Figure 10. exchange Server 2007 topology

VMware, Inc. 3401 Hillview Ave Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www.vmware.com Copyright © 2008 VMware, Inc. All rights reserved. Protected by one or more U.S. Patents Nos. 6,075,938, 6,397,242, 6,496,847, 6,704,925, 6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966, 6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377, 7,111,086, 7,111,145, 7,117,481, 7,149,843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136, 7,277,998, 7,277,999, 7,278,030, 7,281,102, 7,290,253, 7,356,679, 7,409,487, 7,412,492, 7,412,702, 7,424,710, 7,428,636, 7,433,951, 7,434,002 and patents pending. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.