hyper-v: best practices
TRANSCRIPT
OUR SPONSORS MADE THIS POSSIBLE!THANK YOU!
TAKE A DEEP BREATH…
"Design/planning phase" is critical!
HOW MANY…• … hosts?• … CPUs?• … RAM?• … bandwidth?• … ?
• Microsoft Assessment and Planning Toolkit (MAP Toolkit)– https://www.microsoft.com/en-us/download/details.as
px?id=7826– DEMO
HOST (HARDWARE)• use standardized hardware (easier to manage)• use Windows Server certified hardware (https://www.windowsservercatalog.com/)• CPU– server virtualization (Intel VT/AMD-V) and Data Execution Prevention (XD/NX) - Enabled– SLAT-capable with large cache– don’t mix vendors (mixed CPU vendors = NO Live Migration)– Hyper-Threading – enabled– „compatible”• performance impact if not using same instruction set– „best buy” (12-cores per socket?)
• RAM– ECC-capable– the more, the better (how many VMs can go down in case of disaster? 30? )
• HDD (local)– RAID-1 (of two SSDs or HDDs)
POWER OPTIONS• full power or green IT?
• power settings:– BIOS/UEFI level– host OS level– guest OS level
• easy to set:– POWERCFG.EXE /S SCHEME_MIN (High performance)– POWERCFG.EXE /S SCHEME_BALANCED (Balanced
(recommended))
HOST (SOFTWARE) (1)• Hyper-V host is a Hyper-V host (and nothing else)!• install the latest drivers & firmware versions (that work)– use driver/firmware bundles (like SPP for HP servers) to standardize
BIOS, NIC, storage controllers and HBA versions in your environment
• http://h17007.www1.hpe.com/us/en/enterprise/servers/products/service_pack/spp/index.aspx
• install only the necessary (Hyper-V role, Failover Clustering, MPIO + DSM)– use MinShell or Core (or even Nano ) installation options– or full GUI, configure and then „downgrade” to Core– or just full GUI…
• updates & hotfixes– Cluster – https://support.microsoft.com/en-us/kb/2920151– Hyper-V – https://support.microsoft.com/en-us/kb/3135020– support will first ask if you’ve installed all the latest and greatest…
HOST (SOFTWARE) (2)• install backup and monitoring agents• install antivirus if necessary (company policy)– don’t forget to set exclusions!
• don’t forget to modify Hyper-V settings after the installation– default paths are on the system drive!• Set-VMHost -ComputerName localhost -VirtualHardDiskPath 'D:\VMs'• Set-VMHost -ComputerName localhost -VirtualMachinePath 'D:\VMs'
• sign out from Hyper-V hosts when finished working– we are admins… we love PowerShell… we don’t need RDP
sessions to hosts • remember to close remote PS sessions as well
ANTIVIRUS EXCLUSIONS• antivirus exclusions for Hyper-V hosts & clusters are:– all directories that contain virtual machines, their disks, snapshots, …– processes:• %systemroot%\System32\vmms.exe• %systemroot%\System32\vmwp.exe• %systemroot%\Cluster\clussvc.exe• %systemroot%\Cluster\rhs.exe– folders:• %programfiles%\Microsoft\Hyper-V\*• %public%\Documents\Hyper-V\*• %systemdrive%\ClusterStorage\*• %systemdrive%\ProgramData\Microsoft\Windows\Hyper-V\*• %systemroot%\Cluster\*• (witness disk)– https://support.microsoft.com/en-us/kb/3105657 & https://support.microsoft.com/en-us/kb/961804
INSTALLATION• manual or automatic installation• speed-up the deployment– unattend.xml (one-time prepare, many-time use )– DEMO– bare-metal host deployment (SCVMM)– MDT/SCCM deployments
• which version?– Standard, Datacenter (AVMA)
• which option?– full GUI, MinShell, Core, Nano?– full GUI then „downgrade” to Core?• Remove-WindowsFeature Server-Gui-Shell, Server-Gui-Mgmt-Infra
MANAGEMENT TOOLS• although „real (wo)men” use PowerShell for everything, there are
some other tools that can help you: ()– Server Manager console• Best Practices Analyzer (BPA) – useful tool!– Hyper-V console• don’t use this one for managing clustered VMs… please – Failover Cluster console• Cluster Validation Wizard – VERY useful tool! (ask Microsoft
Support! )– System Center VMM• it works… just fine… but not every time… ()– 3rd party solutions (5nine, SysInternals, …)
VIRTUAL MACHINES (1)• use/convert to Generation 2 VMs where possible– https://blogs.technet.microsoft.com/jhoward/201
3/11/14/hyper-v-generation-2-virtual-machines-part-10/
– Convert-VMGeneration tool• https://code.msdn.microsoft.com/ConvertVMGeneration
• use only supported guest OSes/versions– https://technet.microsoft.com/en-us/library/dn79
2027(v=ws.11).aspx• update Integration Components regularly– some OS hotfixes bring new versions of ICs as
well!
VIRTUAL MACHINES (2)• don’t use screen savers inside VMs and sign out of
VMs• templates – use the „smarter” sysprep:– %systemroot%\Sysprep\Sysprep.exe /OOBE /Generalize /Shutdown /Mode:VM– faster sysprep, a lot less hardware recognizing– don’t use this templates for multiple virtualization platforms– Convert-WindowsImage• https://gallery.technet.microsoft.com/scriptcenter
/Convert-WindowsImageps1-0fe23a8f– offline patching – Apply-WindowsUpdate• https://gallery.technet.microsoft.com/Offline-Serv
icing-of-VHDs-df776bda
VIRTUAL MACHINES (3)• devices:– don’t use Legacy Network Adapter if it’s not absolutely necessary– the same goes for virtual HBAs– remove devices you’re not using
• Automatic Stop Action– do not leave „Save” – change to either Shut Down or Turn Off
• disable unnecessary background „things”:– SuperFetch– Windows Search– Scheduled Tasks (disk defragmentation, …)– Aero (optimize for performance)
ACTIVE DIRECTORY/GPO• should I join Hyper-V hosts to Active Directory domain?– YES (existing or the separate „management” domain – little added
security, but huge management overhead; in highly-secured environments)
– Hyper-V will work if domain is not available (but Live Migration won’t!)• should I disable Windows Firewall?– NO (fine-tune it rather!)– Group Policy is a great tool, and your host is domain-joined already, so…
• GPO – disable RDP Printer Mapping– Computer Confguration | Policies | Administrative Templates | Windows
Components | Remote Desktop Services | Remote Desktop Session Host | Printer Redirection | Do not allow client printer redirection | Enable
VIRTUAL MEMORY/PAGING FILE (PAGEFILE.SYS)
• size – leave it on (automatic) at host level• move it to separate disk– for VM – SCSI disk– exclude this disk from Hyper-V Replica
• you can use the following:$computer = Get-WmiObject Win32_ComputerSystem -EnableAllPrivileges$computer.AutomaticManagedPagefile = $false$computer.Put()$CurrentPageFile = Get-WmiObject -Query "select * from Win32_PageFileSetting where name='C:\\pagefile.sys'"$CurrentPageFile.delete()Set-WMIInstance -Class Win32_PageFileSetting -Arguments @{name="D:\pagefile.sys";InitialSize = 0; MaximumSize = 0}
CLUSTER• Hyper-V cluster is a Hyper-V cluster (and nothing else)!• start with at least 3 nodes• separate network communication– Corporate, Storage, LiveMigration, vSwitch, Backup
• use teaming where it seems appropriate• rename cluster resources (networks, disks) and set network
„roles” & priority in cluster• Cluster Aware Updating– use it if it makes sense and don’t use it with SCCM– http://www.altaro.com/hyper-v/cluster-aware-updating-hyper-
v-basics/
NETWORKING (1)• (re)name your NICs (and NIC teams and vSwitches)– „Local Area Connection 23” is a great name, but…
• use teaming (it’s free and it works!)– should you use Active/Actives?• it depends (on underlying network… switches and
connections among them – possible link saturation)• using 1 NIC in Stand-by is just dumb… or not? – Dynamic and Switch Independent is the way to go!• if you have issues with a load balancer, go with Hyper-V port
• which architecture should you use? (converged or not?)– http://www.hyper-v.nu/archives/hvredevoort/2014/02/definitiv
e-guide-to-hyper-v-r2-network-architectures/
NETWORKING (2)• no need for a dedicated heartbeat network in cluster– all cluster networks communicate „heartbeat” signals
• don’t disable VMQ… if you don’t have problems with it (Broadcom?)– http://www.dell.com/support/article/us/en/04/SLN132131
• create virtual switches– … and name them the same (use the convention that works
for you, but same on all hosts)! – how many vSwitches do I need?• smallest number that makes sense (use VLANs!)• (how many NICs? teamed or not? )
NETWORKING (3)• networking optimizations:– set the right networking order – Corporate network first
(ncpa.cpl – Advanced – Advanced Settings – Adapters and Bindings)– disable unnecessary network services• Live Migration – leave only Client for Microsoft Networks, File and
Printer Sharing, TCP/IPv4 and TCP/IPv6 (NetBIOS is disabled)• iSCSI – leave only TCP/IPv4 and TCP/IPv6 (NetBIOS is disabled)– choose which networks will register in DNS (Corporate only)– set cluster network priorities and roles– choose Live Migration network(s)– iSCSI network – use MPIO, not teaming… on separate NICs
NETWORKING (4)• DHCP guard (filtering unsolicited DHCP Server offers)– enable for all VMs, and disable for the ones that need it– small performance impact
• Router Guard (filtering ICMP Router Advertisements and redirect messages)– not used so much
• implement IPAM for addressing– instead of those old, boring Excel files
• do not share adapter with physical OS• enable Jumbo Frames for CSV, iSCSI and LM networks
STORAGE (1)• single LUN per VM?– no need… CSVs work just fine
• SAN or SOFS/SMB3?– if there is SAN in your environment (and it’s supported for Hyper-V),
use it– if you want a „cheaper” solution (and there is no reason against it),
try using SOFS/SMB3– or a combo (SOFS for SMB3 on SAN LUNs)?
• use 4K native disks and 64K allocation unit size for drives hosting VHD(X)
• avoid vIDE for data disks (Generation 1 VMs)• use virtual HBAs only if needed• use Storage QoS if you need it
STORAGE (2)• should I use the „witness” disk in cluster? Yes.• install MPIO, latest DSMs (usually requires restart) and drivers• rename the CSV „mount point” (C:\ClusterStorage\Volume1)”• put VHD(X)s on CSV into folders!– access-related errors if there are VHD(X)s in a CSV root!
• NTFS or ReFS?– NTFS – 64k for VHD(X)s for optimal performance– ReFS – with Windows Server 2016
• measure IOPS– http://blog.workinghardinit.work/2014/01/08/how-to-measure
-iops-of-a-virtual-machine-with-resource-metering-and-measurevm/
VIRTUAL DISKS• use/convert to VHDX• fixed/dynamic/differencing/pass-through?– VHDX – dynamic– VHD – fixed-size
• don’t forget to ensure enough free space for dynamic disks!– the same goes to using thin-provisioned LUNs!
• one-liner:– Convert-VHD -Path D:\VM01.vhd -DestinationPath D:\VM01.vhdx -VHDType Dynamic
• don’t use pass-through disks (no point anymore – dynamic disks are as performant as it gets, and their mobility/manageability is not questionable
• convert pass-through disk to VHDX– New-VHD -Path "D:\VMS\Converted.vhdx" -Dynamic –SourceDisk 5
CLUSTER SHARED VOLUMES (1)• how many and how big?– minimum of 2 (to utilize two different storage controllers, …)– < 8 cluster nodes = 1 CSV per 1 node– > 8 cluster nodes = 1 CSV per 2-4 nodes
• how many VMs per CSV?– up to 50 (server VMs)– up to 100 (client VMs in VDI environment)
• IOPS are all that matters!• enable CSV cache to improve read operations– (Get-Cluster).BlockCacheSize = 2048
• metadata changes can only occur on the CSV coordinator (owner node)• please… rename CSVs before you use them!– C:\ClusterStorage\Volume1 and C:\ClusterStorage\Volume2 are great names,
but…
CLUSTER SHARED VOLUMES (2)• set the automatic stop action to anything other than „Save”• fill up your CSVs with a maximum of 75 percent of its capacity
to allow growth of all these files• how much space is available on your CSVs?– http://www.powershellmagazine.com/2014/02/28/reporting-cl
uster-shared-volume-csv-disk-space-utilization/• encrypt CSV with BitLocker – performance impact of 20-30%• network adapters used for CSVs should have the Client for
Microsoft Networks and File and Printer Sharing for Microsoft Networks enabled– activate Microsoft Failover Cluster Virtual Adapter Performance Filter
(not if you are using Guest Cluster in your virtual machine – then it should be disabled on the host level)
LIVE MIGRATION (1)• don't forget to set the live migration network and
options• use Kerberos constrained delegation instead of
CredSSP (need to log on to the server to start live migration)
• enabling KCD for two node cluster is quite easy ():– HYP1 delegates credentials to HYP2• $hvhost = "HYP2"• $domain = "sun.tklabs.eu"• Get-ADComputer HYP1 | Set-ADObject -Add @{"msDS-
AllowedToDelegateTo"="Microsoft Virtual System Migration Service/$hvhost.$domain", "cifs/$hvhost.$domain","Microsoft Virtual System Migration Service/$hvhost", "cifs/$hvhost"}
LIVE MIGRATION (2)– HYP2 delegates credentials to HYP1• $hvhost = "HYP1"• $domain = "sun.tklabs.eu"• Get-ADComputer HYP2 | Set-ADObject -Add @{"msDS-AllowedToDelegateTo"="Microsoft Virtual System Migration Service/$hvhost.$domain", "cifs/$hvhost.$domain","Microsoft Virtual System Migration Service/$hvhost", "cifs/$hvhost"}
• enable live migration– Enable-VMMigration -Computername HYP1, HYP2– Set-VMHost -Computername HYP1, HYP2 -
VirtualMachineMigrationAuthenticationType Kerberos
• yes, it can be done through GUI as well!
GUEST CLUSTERING• use Anti-Affinity rules to make sure nodes don’t end up
on the same physical host– either VMM or PowerShell• not available through Failover Clustering console!• (Get-ClusterGroup KULEN1).AntiAffinityClassNames = "GuestClusterKULEN"• (Get-ClusterGroup KULEN2).AntiAffinityClassNames = "GuestClusterKULEN" • Get-ClusterGroup KULEN1 | fl Anti*
• change default failover-triggering heartbeat times (to allow live migrations on host, without failover actions in guest – default is 10 seconds)– (Get-Cluster).CrossSubnetThreshold = 25– (Get-Cluster).SameSubnetThreshold = 25
CHECKPOINTS• Windows Server 2012 R2– don’t use checkpoints (snapshots) in production
• Windows Server 2016– use Production checkpoints
• tips:– use them as rarely as possible– checkpoints are no substitution for backups– delete checkpoints as soon as possible– don’t delete a checkpoint file on the file level!• console/PowerShell is the only way to go– use them with caution on domain controllers and database servers
BACKUP AND DISASTER RECOVERY• backup– poor man’s backup solution – Windows Server
Backup– other solutions like SC DPM, Veeam, Altaro, …
• no real need to backup Hyper-V hosts– except maybe for faster recovery, but…
• use Hyper-V Replica if you need disaster recovery– it’s included anyway– exclude Paging file disks from replication
PERFORMANCE (PERFMON.EXE) (1)• storage– it’s all about the IOPS!
– counters:• \Logical Disk(*)\Avg. Disk sec/Read• \Logical Disk(*)\Avg. Disk sec/Write
– thresholds:• up to 15 ms should be OK• 15-25 ms may cause negative impact on workloads• > 25 ms will cause negative impact on workloads
PERFORMANCE (PERFMON.EXE) (2)• memory– there has to be enough memory available inside a VM (or a Hyper-V
host)!
– counters:• \Memory\Available Mbytes– RAM available for running the active processes– 15% (or more) is OK• \Memory\Pages/sec– how often the disk is accessed to resolve hard page faults– < 500 pages per second is OK– > 500 pages per second – machine just needs more RAM (don’t be
cheap!)
PERFORMANCE (PERFMON.EXE) (3)• network– counters• \Network Interface(*)\Bytes Total/sec– shows current network utilization– 20% (or more) free is OK• \Network Interface(*)\Output Queue Length– shows latency in sending network packets (threads that wait on the
NIC)– 0 is OK– 1 or more is a sign of degraded network performance• \Hyper-V Virtual Network Adapter(*)\Bytes/sec– shows which virtual network adapters are consuming the most
bandwidth
PERFORMANCE (PERFMON.EXE) (4)• processor– counters• \Processor(*)\% Processor Time– overall CPU utilization (at host level)– < 80% is always OK () • \Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time– at host level, but to evaluate guest utilization• \Hyper-V Hypervisor Root Virtual Processor\% Total Run Time– CPU of the root partition• \Hyper-V Hypervisor Virtual Processor(_Total)\% Total Run Time– total time of the virtual processors• \Hyper-V Hypervisor Virtual Processor(*)\%Guest Run Time– CPU of the guest partitions– always measure the CPU usage of the physical system by using the Hyper-V
Hypervisor Processor performance counters
PAL IS YOUR NEW (BEST) PAL!• PAL (a.k.a. Performance Analysis of Logs Tool)– reads a performance monitor counter logs and analyzes
them using known thresholds– for Hyper-V, SQL, IIS, Exchange, SharePoint, …– you can get it at https://pal.codeplex.com/– free!
– (P.S. use en-US regional formatting– (P.P.S. Set-Culture en-US )– DEMO
A FEW MORE THINGS...• vCPU allocation (Microsoft recommendation)– no more than 8 virtual CPUs per physical CPU core for server workloads– no more than 12 virtual CPUs per physical CPU core for VDI workloads
• be careful with thin-provisioned disks (running out of storage space), dynamic memory (don’t use it for SQL or Exchange)
• Generation 1 VMs – create bigger VHD(X) and smaller partition– you don’t need to shutdown VM to resize
• don’t forget common tools like defrag and chkdsk• set up a naming convention (and stick to it )• cluster log is your friend… when things go bad (increase sensitivity logging
level)– (Get-Cluster).ClusterLogLevel = 5– Get-ClusterLog -Destination C:\Logs -UseLocalTime
• DOCUMENT EVERYTHING.
RESOURCES• checklists:– https://blogs.technet.microsoft.com/askpfeplat/2013/11/03/windows-serve
r-2012-r2-hyper-v-best-practices-in-easy-checklist-form/– https://blogs.technet.microsoft.com/askpfeplat/2013/03/10/windows-serve
r-2012-hyper-v-best-practices-in-easy-checklist-form/• book:– https://www.packtpub.com/virtualization-and-cloud/hyper-v-best-practices
• other:– http://www.showit.sk/ShowIT/media/ShowIT/prezentacie/2015/marek_jan_
Microsoft-Hyper-V-performance-tuning.pdf– http://www.altaro.com/hyper-v/23-best-practices-improve-hyper-v-vm-perf
ormance/– http://www.altaro.com/hyper-v/19-best-practices-hyper-v-cluster/– http://www.altaro.com/hyper-v/common-hyper-v-deployment-mistakes/– http://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/
THANK YOU!