Best Practices with Designing Exchange Server

for Virtualized Environments

David Zazzo

Senior Consultant, Microsoft

david.zazzo@microsoft.com

Disclaimer

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Office 365, and other

product and service names are or may be registered trademarks and/or trademarks in the

U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view

of Microsoft Corporation as of the date of this presentation. Because Microsoft must

respond to changing market conditions, it should not be interpreted to be a commitment

on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information

provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES,

EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Exchange 2013 Disclaimer

Performance testing and validation is currently in progress

on Exchange 2013 … the Exchange Product Group has not

yet finalized their guidance on virtualizing E2013.

Guidance may change, keep an eye on TechNet for the

latest on virtualization guidance

About me

• Senior Consultant with Microsoft Consulting Services

• At Microsoft for nearly five years and counting…

• Instructor for Microsoft Certified Master | Exchange

– Virtualization

– Load Balancing

– RBAC

Why Virtualize Exchange?

• Money, Money, Money – Licensing benefits – multiple servers under one license

– Server Consolidation • Power, Space, Cooling

– Maximize Utilization of Hardware • Underutilized processors == too much money spent

• Underutilized memory == too much money spent

• Politics – “We bought it, we’re going to use it everywhere”

– CIOs / managers reading “CIO Magazine” again…

– Senior company leadership

SUPPORTABILITY

What’s supported, what’s not?

Virtualization Host Support Guidelines

• Hypervisor must be one of the following platforms:

– Windows Server Hyper-V (2008, R2, or 2012)

– Hyper-V Server (2008, R2, or 2012)

– SVVP-certified hypervisor

http://www.windowsservercatalog.com/svvp.aspx

• Using Windows Azure, Amazon EC2, etc. is not supported

Virtualization Guest Support Guidelines

• Exchange 2010 Guest OS:

– Guest OS: Windows Server 2008 SP2 or R2 • Exchange Server 2010 SP3 will support 2012, not yet released

– E2010 RTM: All roles except UM

– E2010 SP1+: All roles including UM

• Exchange 2013 Guest OS:

– Guest OS: Windows Server 2008 R2 or Windows Server 2012

– All roles (CAS + MBX)

Virtualization Guest Support Guidelines

• Using Host-Based High Availability (Live Migration,

vMotion, etc)

– Exchange 2010 RTM: Can not mix with DAG

– Exchange 2010 SP1+: Can mix with DAG*

– Exchange 2013: Can mix with DAG*

* You can mix DAG and HA with certain constraints – more on this

later in the presentation

Virtualization Guest Support Guidelines

• Maximum CPU ratio of 2:1 vCPU:pCPU

– Example: If you have 8 cores, maximum supported is 16 vCPUs

assigned to guests

– Recommended ratio is 1:1, don’t overcommit CPU in production

• iSCSI initiator in guest is supported

– Check with your vendor, make sure VM network stack supports

jumbo frames, full network fidelity

What’s Not Supported?

- Dynamic memory & memory over-commit

- Not supported for any Exchange role

- Hypervisor snapshots (Time Travel)

- Not supported for any Exchange role

- Differencing/delta/dynamic disks

- Not supported for any Exchange role

- Other applications installed on the root

- Only deploy your management, monitoring, A/V, etc.

Time Traveling (Hypervisor Snapshots)

• Hypervisor snapshots make lab testing much easier

• Resist the temptation to use hypervisor snapshots in

production – they aren’t supported!

• Aspects of the Exchange system do not handle time travel

well (notably log shipping)

• Use caution with snapshots in the lab (you may need to

roll back more than one VM)

PLANNING FOR EXCHANGE

Planning Guidance

Historically, our planning guidance has been very simple:

Determine your baseline requirements, then have the

physical/virtual discussion. Why?

– You still need enough CPU

– You still need enough memory

– You still need enough disk I/O

– You still need enough network bandwidth

Unified Messaging Virtualization

• Requires Exchange 2010 SP1+ for support

• VM should have a minimum of:

– (4) Virtual Processors/Cores

– Requires 1:1 vCore:pCore allocation

– 16GB of memory

• Unified Messaging server role must stand alone (2010)

– Hub, CAS, and Mailbox can not be in the same VM as Unified Messaging role

VS

MBX

HUB

MBX

CAS

HUB

HUB

HUB

HUB

CAS

CAS

CAS

CAS

MBX

MBX

MBX

MBX

• Use common sense when placing your VMs

• Deploy VMs with the same role across multiple roots

• Do not deploy MBX VMs in the same DAG on the same root server

VM Placement Recommendation (2010)

Host-Based High Availability

What is Host-Based High Availability?

Automatic failover of virtual machines to another virtualization

host in the event of a critical hardware failure (virtualization

platform independent)

Host-Based High Availability Concepts

• Key Concepts

– Cold boot: Bringing a system from a “power-off” state to a clean, fresh

start of the OS

– Saved state: Many hypervisors allow you to save state, or “hibernate” a VM

rather than shutting it down or turning it off. On resume, you’re resuming

from the saved state.

– Planned migration: Using Live Migration or vMotion to move a VM from

one host to another by administrator action.

Host Based High Availability

What you need to be aware of:

• Not an Exchange-aware solution

– No knowledge of transaction logs, clean/dirty database dismount,

database checksums, Exchange-health, etc.

• Only protects against server hardware/network failure

• No protection against storage failure / data corruption

• Trend is larger mailboxes = larger database sizes = longer time to

fully recover from data loss (cache warming)

• Requires the guest VM to perform a cold boot (this is HA?)

Host Based Failover Clustering

What is Supported:

• Automatically failing VMs to an alternate cluster node in the event of a critical

host hardware issue

– Important! VM must come up from a cold start, not saved state or warm start

• Live Migration / vMotion / etc.

What is Not Supported:

• Quick Migration (Windows Server 2008 pre-R2)

• Anything that pauses/saves state, migrates, and then resumes

• “Time Traveling”

Live Migration Considerations

• Adjust cluster heartbeat if necessary

– Target less than 5 seconds for Live Migration / vMotion

– Adjust if Live Migration takes longer than 5 seconds, but no

more than 10 seconds

• Enable jumbo frames on the Live Migration network(s)

• Use very fast networks (5Gb, 10Gb)

Which HA to choose?

• Microsoft recommends Exchange HA (DAG)

– Exchange-aware HA (log shipping, single page restore, no more

-1018s, best copy selection, etc.)

– Understand “failure domains”

• Power, network, storage, rack, blade chassis, etc…

– Make sure your DAG members are placed where you think they

should be, and that they don’t migrate where they shouldn’t be

Disaster Recovery / Backup Considerations

• Design your backup solution carefully!

– Use Exchange-aware VSS solutions in guests to get the Exchange stores

and guest OS instance

– Use VSS on the host to backup the hypervisor and configuration

• Interesting DR scenarios

– Some physical, some virtual

– Ensure that if you have a virtual machine as a target for multiple physical

server in your DR scenario, that it can handle the worst case DR

Hardware Considerations

• Disk subsystem must still support required IOPS

– Jetstress is still the gold standard

• Test as a system

– Jetstress and LoadGen all VMs on the system to stress the entire system

with a realistic peak load

– Just like with physical, one or the other is not enough

– Unless your clients are going to be virtual in production (VMware VDI for

instance), test with physical LoadGen clients

• Watch “over-oversubscribe” of processors

– Remember the vCore : pCore ratio (2:1 max supported, 1:1 recommended)

Hardware Considerations, Con’t

• Network

– Bandwidth

– Take into account iSCSI when necessary! • High bandwidth customers should avoid 1Gb iSCSI

• Newer FC4Gb, FC8Gb, 10Gb iSCSI or FCoE better

• New hardware vs reuse/upgrade of old

– Will older procs support 64-bit guests (watch new procs, too)

– Can you upgrade the procs in an old server to work?

– Will upgrading the memory cost more than a new server?

Deployment Recommendations

• Exchange Server 2010/2013 is not “virtualization aware”

• Core Exchange Design Principles Still Apply – Design for Performance, Reliability and Capacity

– Design for Usage Profiles (CAS/MBX)

– Design for Message Profiles (Hub/Edge)

• Virtualization Design Principles Now Apply – Design for Performance, Reliability and Capacity

– Virtual machines should be sized specific to the Exchange role (EDGE, HUB, CAS, MBX, multi-role)

– Hosts should be sized to accommodate the guests that they will support

• Database Cache requirements are the same for physical and virtual deployments

Total Send + Receive (75k message size)

Database Cache Per Mailbox (MB)

50 3

100 6

150 9

200 12

250 15

300 18

350 21

400 24

450 27

500 30

Mailbox Server Guidelines (2010)

Total Send + Receive (75k message size)

Megacycles per Active Mailbox

Physical MBX Role

Megacycles per Passive Mailbox

Physical MBX Role

Megacycles per Active Mailbox

Virtual MBX Role

Megacycles per Passive Mailbox

Virtual MBX Role

50 1 0.15 1.1 0.17

100 2 0.3 2.2 0.33

150 3 0.45 3.3 0.5

200 4 0.6 4.4 0.66

250 5 0.75 5.5 0.83

300 6 0.9 6.6 0.99

350 7 1.05 7.7 1.2

400 8 1.2 8.8 1.32

450 9 1.35 9.9 1.49

500 10 1.5 11 1.7

• Virtual Processor ≠ Logical Processor

• Hypervisor and the Virtualization Stack consume CPU

• Reduce recommended mailbox count by ~10%

Mailbox Server Guidelines (2010)

Mailbox Server Guidelines (2013)

• Unfortunately, too early … guidance coming soon, watch

TechNet and EHLO

• Expect that the 10% rule of thumb will continue

• Virtual SCSI (passthrough or fixed disk VHD/VMDK) • Recommended configuration for database and log volumes

• iSCSI

• Standard best practice for iSCSI connected storage apply (dedicated NIC, jumbo frames, offload, etc.)

• iSCSI initiator in the guest is supported but need to account for reduced performance

– Clarify support between hypervisor vendor and storage vendor (certain

combinations are not supported)

Mailbox Storage Configuration

Mailbox Server Role Requirements Calculator

Mailbox Server Role Requirements Calculator

• The ‘storage calculator’ – but much, much more now!

• The ‘de facto’ Exchange calculator

• Created during Exchange 2007 product cycle

• Properly calculates host CPU, memory, storage

requirements, network bandwidth for replication

• Newer versions generate database/dag deployment

scripts

Using the Calculator to Plan for Virtualization

Get the Exchange Processor Query Tool

– http://msexchangeteam.com/archive/2010/10/27/456738.aspx

This allows you to put in a processor and number of cores,

and have the tool give you the average SpecInt value you

can use in the calculator

Using the Calculator to Plan for Virtualization

X/(N*Y) = per virtual processor SPECInt2006 Rate value

Where X is the SPECInt2006 rate value for the hypervisor host server

Where N = the number of physical cores in the hypervisor host

Where Y = 1 if you will be deploying 1:1 virtual processor-to-physical

processor on the hypervisor host

Where Y = 2 if you will be deploying up to 2:1 virtual processor-to-

physical processor on the hypervisor host

Getting the Calculator to Plan for Virtualization

• Latest Mailbox Server Role Requirements Calculator

(v17.2) now includes support for virtualization!

• Hypervisor CPU Adjustment Factor is the virtualization

overhead

(rule of thumb is 10%)

WHY NOT VIRTUALIZE?

Now that we’ve talked about virtualizing Exchange…

Why NOT Virtualize Exchange?

• Virtualization does add complexity

• Virtualization does impact performance

• You do still have to manage those virtual servers as if they were physical

– Licensing, Hotfixes, Service Packs, Updates, Monitoring, etc.

• You also have to “manage” that virtualization platform

• Are there alternatives? Do they make sense?

– Can a physical architecture make full use of hardware?

– Would use of blade server technologies make more sense than virtualization?

The Alternative: Exchange Multi-Role Server

• Current Microsoft guidance is to deploy multi-role servers

(CAS, HUB, MBX) instead of virtualization

• Your “core math” works out:

– Example 8-core Physical Machine

– Mailbox Server Role gets 4 cores

– Client Access Server Role gets 3 cores (3:4 sizing)

– Hub Transport gets 1 core (1:5 sizing, this is better)

• Take full advantage of the physical machine

Additional reading…

• Windows Server 2008 R2 Hyper-V http://www.microsoft.com/windowsserver2008/en/us/hyperv-main.aspx

• Windows Virtualization Team Blog http://blogs.technet.com/virtualization

• Infrastructure Planning and Design Guides for Virtualization http://technet.microsoft.com/en-us/solutionaccelerators/ee395429.aspx?SA_CE=VIRT-IPD-WEB-MSCOM-2009-09-21

• Exchange Server 2010 Guidance http://technet.microsoft.com/en-us/library/bb124558(EXCHG.140).aspx

• Exchange Team Blog http://blogs.technet.com/exchange

• Best Practices Whitepaper for Virtualizing Exchange 2010 http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8647c69d-6c2c-40ca-977e-18c2379b07ad

QUESTIONS?

VAŠA VPRAŠANJA?

Hvala za udeležbo in

prosim izpolnite ankete.

Thank you!