APP-BCA1684

Virtualizing
Exchange Best
Practices
Scott Salyer
VMware, Inc.

#vmworldapps

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in
any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features
discussed or presented have not been determined.

Agenda

Exchange Design on vSphere

Support Considerations
vSphere Best Practices for Exchange

vSphere HA, DRS and vMotion with Exchange

Site Resiliency for Exchange

Exchange Design Process

Set aside virtualization for a momentremember, were designing

an Exchange environment first

Complete pre-requisite work before diving into the design:

Understand the business and technical requirements
Evaluate the current workload, or estimated workload for a new environment
Know the health of the current and supporting infrastructure
Understand support and licensing considerations

Follow a design methodology:

1. Establish the design requirements
2. Gather the compute requirements based on the Exchange design
requirements (Exchange Role Requirements Calculator)

3. Design the virtualization platform based on the Exchange design and

compute requirements

4. Determine virtual machine sizing and distribution among virtualization hosts

4

Design Requirements

High availability requirements

Database Availability Groups (DAG)
vSphere High Availability

Site resiliency requirements

DAG
Site Recovery Manager

Dedicated or Multi-role servers

Database Sizing
Few, large databases or many, small databases
Consider backup and restore times

Number of mailboxes
Growth over next X years

Mailbox tiers
Quota, activity, deleted item retention, etc
5

Compute Requirements based on Exchange Design

Requirements
1 site, high availability using DAG and vSphere HA
10,000 heavy users (2GB quota, 150 messages/mbx/day)
Dedicated mailbox role, combined client access and hub transport servers

*CPU recommendations based on Intel x5660 processor

Physical Compute Requirements

(7) mailbox cores to support activated databases
(7) client access/hub transport cores

vSphere Compute Requirements

Exchange compute requirements

14 total mailbox cores to support failover
14 total client access/hub transport cores to support failover
256 GB memory for mailbox role to support failover (assuming two node DAG)
48 GB memory for CAS/HT role to support failover (assuming four CAS/HT VMs)
No. of CAS/HT VMs * (4 GB + (2 GB * No. of vCPUs))

Sample vSphere host configuration

CPU: (2) six-core Intel x5660
Memory: 144 GB

vSphere minimum requirements

Three vSphere hosts provide 36 cores, 432 GB

Virtual Machine Sizing and Placement

Option 21:
(3)
(2) DAG nodes (54%
(77% utilized during failover)
6
8 vCPU, 64GB
128GB

(4) client access/hub transport nodes

4 vCPU, 12GB (2GB/vCPU + 4GB)

Sufficient
CPU and Memory
resources
overcommitted
for peripheral during
services
host failure

Support Considerations (What is and what isnt?)

Support for Exchange has evolved drastically over the last two
years leading to confusion and misconceptions

What is Supported?
Virtualization of all server roles, including Unified Messaging with Exchange
2010 SP1

Combining Exchange 2010 SP1 DAG and vSphere HA and vMotion

Thick virtual disks and raw-device mappings (pass-thru disk)
Fibre channel, FCoE, iSCSI (native and in-guest)

Not Supported?
NAS Storage for Exchange files (mailbox database, HT queue, logs)
Thin virtual disks
Virtual machine snapshots (what about backups?)
MS TechNet Understanding Exchange 2010 Virtualization:
(http://technet.microsoft.com/en-us/library/jj126252)
9

Virtual CPUs

Best Practices for vCPUs

vCPUs assigned to all Exchange virtual machines should be
equal to or less than the total number of physical cores on the
ESX host
Enable hyper-threading, but understand that logical processor
cores are not equal to physical processor cores for sizing
Enable NUMA -- Exchange is not NUMA-aware, but ESX is
Determine the CPU requirements based on physical CPU
throughput and mailbox profile
Use the SPECInt2006 rating for proposed processor
Exchange Processor Query Tool
Calculate adjusted megacycles required and match up to proposed hardware
Exchange Mailbox Role Calculator

10

Hyper-threading Confusion

VMwares Performance Best Practices for vSphere whitepaper

recommends enabling hyper-threading in the BIOS
http://www.VMware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf (page 15)
www.VMware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf (page 20)

Microsoft recommends disabling hyper-threading for production

Exchange implementations
http://technet.microsoft.com/en-us/library/dd346699.aspx#Hyper
Hyper-threading causes capacity planning and monitoring challenges, and as
a result, the expected gain in CPU overhead is likely not justified

So, do I or dont I?

YES! Enable hyper-threading to take advantage of ESXis

intelligent scheduler, but size to the capability of physical cores
11

Sizing Exchange VMs to the NUMA Node

4 Core / 32 GB

4 Core / 32 GB

NUMA Node 1

NUMA Node 2

NUMA Interconnect

Memory Bank 1

4 vCPU/32 GB
Within NUMA Node

Memory Bank 2
6 vCPU/48 GB
Beyond NUMA Node

12

NUMA Considerations
The following recommendations should be followed whenever possible to
ensure the best performance on systems that support NUMA:

Make sure the NUMA architecture is enabled in the BIOS. On many systems
enabling NUMA is achieved by disabling Node Interleaving (typically default)

When possible size Exchange virtual machines so that the amount of vCPUs
and memory does not exceed the number of cores and memory in a single
NUMA node

The size of a NUMA node is not always the number of cores on a chip, for
example; AMD Magny Cours place two six-core chips in a single die/socket

Avoid using CPU affinity features in vSphere, as this can circumvent the NUMA
architecture and reduce performance.

13

Virtual Memory
Best Practices
Avoid memory over-commitment
Only use memory reservations to avoid over-commitment, guarantee
available physical memory, or to reclaim VM swap file space.
Right-size the configured memory of a VM.

More memory means less IOs, but

only slightly. Test in your
environment and fine-tune.
Check Unlimited in resource
allocation or set the limit to higher
than configured memory

14

Storage: Best Practices

Follow storage vendor recommendations for path policy;

no restrictions like with MSCS

Format NTFS partitions from within the guest with a 64KB

allocation unit size

Verify partition is aligned

StartingoffSet

StripeUnitSize

Eagerzeroedthick virtual disks eliminate first write penalty

Do not use if using array thin provisioning

Set power policy to High Performance

Or disable power management in BIOS

15

Storage: Common Questions

PVSCSI vs LSI Logic

PVSCSI can provide better throughput with less CPU utilization, test using
JetStress and your proposed workload

More virtual SCSI adapters

Evenly distribute virtual disks and

RDMs across four vSCSI adapters

VMDK or RDM?
Next Slide

One virtual disk per VMFS or

consolidated virtual disks?
Performance VMFS datastore must be sized
to accommodate the combined workload

Management Fewer datastores eases

VMFS

management
VMFS

16

VMFS

VMFS

When should I use raw-device mappings (RDMs)?

Performance

Performance is no longer a deciding factor for using RDMs

VMDK disks perform comparably to RDMs
Capacity

VMDK files are limited to 2TB

Physical mode RDMs support up to 64TB
Storage Interaction

Backup solutions may require RDMs due to storage interaction needed

for hardware based VSS
Considerations

Easier to exhaust 255 LUN limitation in ESXi

VMFS volumes can support multiple virtual disks
vSphere storage features leverage virtual disks
17

What about NFS and In-guest iSCSI?

NFS
Not supported for Exchange data (databases or logs) or shared-disk MSCS
configs (it works great, but consider support implications)

Consider using for guest OS and app data

In-guest iSCSI
Supported for DAG database storage
Facilitates easy storage zoning and access masking
Useful for minimizing number of LUNs zoned to an ESXi host
Offloads storage processing resources away from ESXi hosts

18

Networking
Best Practices
vSphere Distributed Switch or Standard vSwitch?
Choice is yours, but distributed switches require less management overhead

Separate traffic types:

Management (vmkernel, vMotion, FT)
Storage (iSCSI, FCoE)
Virtual Machine (MAPI, replication, DMZ, etc)

Configure vMotion to use multiple NICs to increase throughput

Allocate at least 2 NICs per virtual switch to leverage NIC teaming
capabilities
Use the VMXNET3 para-virtualized network interface within the guest
Following Microsoft best practices, allocate multiple NICs to Exchange
VMs participating in a DAG

19

Exchange DAG Networking

DAG VMs *should* have two virtual network adapters for

replication and client access (MAPI)
If separate networks are not possible use a single virtual NIC

Single vSwitch
using VLAN
trunking to
separate traffic

20

Physical
Separation using
multiple vSwitches
and physical NICs

vSphere High Availability

Best Practices

Enable HA to protect all VMs in the case of an unexpected

host failure

Configure Admission Control policy to ensure failover capacity

is available

Enable VM Monitoring to restart

non-responsive VMs

Protect database availability groups

with vSphere HA
Deploy vSphere clusters as N+1, where N
is the number of nodes in a DAG

21

VMware Distributed Resource Scheduling

Best Practices

Enable DRS in fully automated mode for entire cluster

Set individual automation levels for one-off needs

When possible keep VMs small

VMs with less configured memory and fewer vCPUs can be placed by DRS
more efficiently

vSphere cluster members should have compatible CPU models for

vMotion compatibility
EVC Mode can guarantee host compatibility

Use host and VM groups and affinity rules

22

DRS VM and Host Groups and Rules

Host DRS Groups Use to group like hosts; hosts in a rack or

chassis, or for licensing purposes

VM DRS Groups Use to group like or dependent VMs together;

all CAS/HT VMs, all DAG nodes, etc.

Rules:

Keep VMs Together

Separate VMs

VMs to Hosts

23

Use for DAG VMs

Must run on hosts in group

Should run on hosts in group
Must not run on hosts in group
Should not run on hosts in group

Use for non-clustered

VMs

Should Run On Versus Must Run On Rules

Should run on rules preferred to

keep like VMs separated across
chassis, racks, datacenters, etc.

In the case of a failure, VMs can be

powered on surviving hosts to
maintain performance

Must run on rules preferred when a

VM must never run on a particular
host or set of hosts

In the case of a failure, VMs will NOT

be allowed to run on the specified
hosts unless the rule is removed,
even if vCenter Server is offline

24

Avoid Database Failover during vMotion

When using vMotion with DAG nodes
If supported at the physical networking layer enable jumbo frames on all
vmkernel ports to reduce the frames that must be generated and
processed

If jumbo frames is not supported modify cluster heartbeat setting

samesubnetdelay parameter to a maximum of 2000ms (default = 1000ms)

C:\> C:\cluster.exe /cluster:dag-name /prop

samesubnetdelay=2000
PS C:\> $cluster = get-cluster dag-name;
$cluster.SameSubnetDelay = 2000

Always dedicate vMotion interfaces for the best performance

Always use multiple vMotion interfaces for increased throughput
25

Multi-NIC vMotion Support

vDS management port group configured with both vmnics active

26

Multi-NIC vMotion Support

vDS vMotion port group 1 configured with vmnic0 active, vmnic1 standby

27

Multi-NIC vMotion Support

vDS vMotion port group 2 configured with vmnic0 standby, vmnic1 active

28

Multi-NIC vMotion Support

Configure in teaming properties of port group

29

Backups

Virtual Machine Backups

Requires Exchange-aware agent to ensure a supported Exchange backup
Large virtual machines (many virtual disks) take longer to snapshot and
commit may affect DAG heartbeats

Software VSS
Wide third party adoption
Flexible configuration support

Hardware VSS
Storage array level protection, either full clones or snapshots
Storage vendor provides VSS integration
Most solutions require physical mode raw-device mappings (unless using inguest attached iSCSI)

30

Site Resiliency Exchange Native

Deployment may involve two or more sites serving as failover targets for
each other

Passive databases located in primary and secondary sites provide local

high availability and site resilience
Datacenter switchover process:
1. Terminate services in primary site
(stop-databaseavailabilitygroup)
2. Validate dependencies in second
datacenter

3. Activate mailbox servers

(restore-databaseavailabilitygroup)
4. Update DNS records for client access
endpoints

31

Site Resiliency Site Recovery Manager

Deployment may involve two or more sites serving as failover targets for
each other
Passive databases at the primary site provide high availability, storage
replication provides site resiliency
SRM failover process:
1. Initiate SRM recovery plan (also
updates DAG node IP addresses)
2. Configure DAG IP address and
witness server (may be part of
SRM recovery plan)
3. Update DAG and DAG node A
records (may be part of SRM
recovery plan)
4. Update DNS records for client
access endpoints (may be part of
SRM recovery plan)

32

Choosing a Site Resilience Plan

Exchange Native Site Resilience

Active-active and Active-passive deployments supported
Manual activation during site failure
Requires application specific site recovery procedure
Testing recovery procedure requires failing over production services

Site Recovery Manager

Active-active and Active-passive deployments supported
Manual activation during site failure
Application independent recovery
Testing of recovery plans can be done without impacting production

33

Take Aways

100% support from VMware and Microsoft

Virtualization doesnt change the compute requirements for
Exchange

Design the Exchange environment based on requirements,

base vSphere design on Exchange design
Visit our Exchange Page on VMware.com
http://www.vmware.com/go/exchange

VMware Communities: Exchange, Domino and RIM

34

FILL OUT
A SURVEY
AT WWW.VMWORLD.COM/MOBILE

COMPLETE THE SURVEY

WITHIN ONE HOUR AFTER
EACH SESSION AND YOU WILL
BE ENTERED INTO A DRAW
FOR A GIFT FROM THE
VMWARE COMPANY STORE

APP-BCA1684

Virtualizing
Exchange Best
Practices
Scott Salyer
VMware, Inc.

#vmworldapps