Technical white paper

Implementing Windows Server 2012 SR-IOV on HP ProLiant Servers and HP ProLiant server blades, 2nd edition

Table of contents Introduction ............................................................................................................................................................................ 2

Overview of SR-IOV ............................................................................................................................................................... 2

Windows Server 2012 recommended system configurations ........................................................................................ 3

SR-IOV-compliant ProLiant servers .................................................................................................................................... 3

SR-IOV-compliant NICs ......................................................................................................................................................... 4

Enabling the SR-IOV feature ................................................................................................................................................ 5

Prepare Materials .............................................................................................................................................................. 5

Configure System .............................................................................................................................................................. 6

Update the Intelligent Provisioning firmware ............................................................................................................... 7

Install Windows Server 2012 ........................................................................................................................................... 7

Installing the HP SPP ........................................................................................................................................................ 7

Configure Server Manager ............................................................................................................................................... 8

Enable SR-IOV on a Virtual Machine ............................................................................................................................ 11

Making sure that the VM is enabled ................................................................................................................................. 12

Recommendations for guest OS configurations ............................................................................................................ 13

Troubleshooting the VM enablement issues .................................................................................................................. 13

Known issues and workarounds ...................................................................................................................................... 14

Resources, contacts, or additional links .......................................................................................................................... 16

Click here to verify the latest version of this document

Technical white paper | Microsoft Windows Server 2012 SR-IOV

2

Introduction

Microsoft® Windows® Server 2012 supports Single Root I/O Virtualization (SR-IOV) as a new feature for Hyper-V. SR-IOV is included in a Peripheral Component Interconnect Special Interest Group (PCI-SIG) specification for partitioning PCI Express (PCIe) adapter bandwidth. Windows Server 2012 supports SR-IOV through SR-IOV capable PCIe network adapters (NICs).

Partitioning port bandwidth provides several benefits, including:

• Increased I/O data throughput

• Reduced CPU utilization for processing network traffic

• Reduced network path latency

• Support for live migration

For more information on the SR-IOV standard and industry support for the standard, go to the PCI-SIG site at: pcisig.com.

This paper focuses on implementing SR-IOV within Windows Server 2012. The paper includes the requirements and recommendations for using the Windows Server 2012 SR-IOV feature on HP ProLiant Servers and HP ProLiant server blades.

Before implementing SR-IOV, first make sure that your server meets the Windows Server 2012 configuration requirements as established by Microsoft. For details on these requirements, see the Microsoft website at microsoft.com/library/jj134246.aspx.

For detailed information or to download Windows Server 2012, see the Microsoft Windows Server 2012 home page at microsoft.com/en-us/server-cloud/windows-server/default.aspx.

Overview of SR-IOV

SR-IOV allows multiple VMs to share a single SR-IOV-capable PCIe NIC while retaining the performance benefit of one PCIe device to one VM association.

The SR-IOV standard builds on the following PCI standards to extend the PCI configuration space and define access to Virtual Functions:

• PCIe

• Alternative Routing ID (ARI)

• Address Translation Services (ATS)

• Function Level Reset (FLR)

By assigning a Virtual Function (VF) to each VM, multiple VMs can share a single SR-IOV capable PCIe NIC that may have just one physical network port.

A Virtual Function is a function that has an association with a Physical Function. Each partition of the PCIe NIC appears in the PCI configuration space as a Virtual Function with its own configuration space and Base Address Registers (BAR). It shares the bandwidth of the PCIe NIC with other Physical Functions or Virtual Functions. Characteristics of a Virtual Function include:

• Support for native I/O virtualization

• A PCIe configuration space that defines a unique set of Message Signaled Interrupts (MSI) and MSI-eXtended (MSI-X) vectors

• A Physical Function is a PCIe function that supports the SR-IOV capabilities. It is a standard PCIe device with its own full PCI configuration space and set of resources to manage Virtual Functions. Characteristics of a Physical Function include:

• SR-IOV Extended Capability as part of its configuration space

• Configuration information for the Virtual Functions to which it is associated

• Base Address Registers (BAR) configuration and Virtual Function type for any Virtual Functions associated with it

SR-IOV implements bandwidth partitioning by allocating multiple Virtual Functions to a Physical Function. The Virtual Functions appear as PCI devices that are part of the physical PCI device by their resources (queues and register sets). Instead of using the Virtual Machine Manager for VM processing, SR-IOV bypasses the VMbus with dedicated hardware queues and a round-robin scheduler to reduce processor overhead.

Because SR-IOV enumerates Virtual Functions in a hardware-based PCI configuration space, it is well suited for providing hardware support for virtualization of fixed-function devices. A Virtual Function and Physical Function that are associated

Technical white paper | Microsoft Windows Server 2012 SR-IOV

3

must be of the same type because the Virtual Function inherits its Vendor ID and Device ID from its associated Physical Function’s configuration space fields.

Windows Server 2012 recommended system configurations

The PCI subsystem and the Windows Server 2012 kernel contain the core SR-IOV implementation, so both must support SR-IOV. In addition, the PCIe NIC driver must support Virtual Functions and Physical Functions.

Processor virtualization (such as AMD-v and Intel® VT-x) and chipset virtualization (such as AMD-Vi and Intel VT-d) allow a VM and guest OS to share a SR-IOV-capable PCIe NIC. This allows the Virtual Machine Manager to assign one or more Virtual Functions to each VM.

In addition to the Windows Server 2012 configuration requirements, your server configuration must also meet the following criteria to use SR-IOV:

• The processor must support Second Level Address Translation (SLAT) and I/O Memory Management Unit (IOMMU) technology. To meet these requirements, we recommend using either of the following processors. See the “SR-IOV-compliant ProLiant servers” section for details on our servers that include these processors and support Windows Server 2012 SR-IOV:

– Intel Xeon® Processors with Intel64 architecture with VT-x and EPT

– Advanced Micro Devices, Inc. (AMD64) Opteron Series Processors with AMD-Vi and NPT

• The System ROM must be capable of enabling and describing the SR-IOV capabilities to the Windows Server 2012 operating system

• You must use an SR-IOV–capable NIC and driver

SR-IOV-compliant ProLiant servers

Table 1 lists the ProLiant servers and minimum System ROM version that are required for Windows Server 2012 and the SR-IOV feature.

Important

SR-IOV requires both hardware and System ROM compliance to work. Your server must have the minimum System ROM version listed in the table to enable SR-IOV. If necessary, update the System ROM before beginning enabling SR-IOV.

You can download System ROM updates from the HP Service Pack for ProLiant (HP SPP), which is available at hp.com/go/spp/download or the HP Support Center at hp.com/go/support.

Table 1. Required System ROM versions

Server platform System ROM family System ROM Version

ProLiant BL servers

BL420c Gen8 I30 8/20/2012 or later

BL460c Gen8 I31 8/20/2012 or later

BL465c Gen8 A26 8/20/2012 or later

BL660c Gen8 I32 8/20/2012 or later

ProLiant DL servers

DL360e Gen8 P73 8/20/2012 or later

DL360p Gen8 P71 8/20/2012 or later

DL380e Gen8 P73 8/20/2012 or later

Technical white paper | Microsoft Windows Server 2012 SR-IOV

4

Server platform System ROM family System ROM Version

DL380p Gen8 P70 8/20/2012 or later

DL385p Gen8 A28 8/14/2012 or later

DL560 Gen8 P77 8/20/2012 or later

DL580 G7 P65 8/4/2012 or later

DL585 G7 A16 8/14/2012 or later

DL980 G7 P66 7/30/2012 or later

ProLiant ML servers

ML350e Gen8 J02 8/20/2012 or later

ML350p Gen8 P72 8/20/2012 or later

ProLiant Blade/SL line servers

SL230s Gen8 P75 8/20/2012 or later

SL250s Gen8 P75 8/20/2012 or later

SL270s Gen8 P75 8/20/2012 or later

SR-IOV-compliant NICs

We support the SR-IOV options listed in Table 2 for Windows Server 2012.

Table 2. Supported ProLiant SR-IOV options

Option Drivers Location

Web download Windows media Windows Server 2012 Supplement for Service Pack for ProLiant

HP Ethernet 10Gb 2-port 560FLB Adapter

HP Ethernet 10Gb 2-port 560SFP+ Adapter

HP Ethernet 1Gb 4-port 366FLR Adapter

HP Ethernet 10Gb 2-port 530FLR-SFP+ Adapter

HP Flex-10 10Gb 2-port 530FLB Adapter

HP Ethernet 10Gb 2-port 530SFP+ Adapter

HP Flex-10 10Gb 2-port 530M Adapter

HP Ethernet 10Gb 2-port 530T Adapter

EVBDA.SYS and

BXND60A.SYS (applies to all)

The drivers for these NICs are available on the Windows Server 2012 media or the HP SPP, which is available at hp.com/go/spp.

We will update this document to include newly supported configurations as our testing progresses.

Technical white paper | Microsoft Windows Server 2012 SR-IOV

5

Enabling the SR-IOV feature

Figure 1 provides an overview of the process for enabling the SR-IOV feature.

Figure 1: SR-IOV enablement process overview

Prepare Materials

As preparation for enabling the SR-IOV feature, make sure that your server and components are supported with Windows Server 2012 and SR-IOV:

• Select a server from the recommended system platforms listed in Table 1 of the “SR-IOV-compliant ProLiant servers” section.

• Make sure the the ProLiant server has the System ROM version indicated in Table 1 of the “SR-IOV-compliant ProLiant servers” section.

• Make sure that the server has a DVD drive (either installed or attached to the server receiving the installation). If you will be using iLO virtual media with either a HP ProLiant DL or ML server, you will need a license key.

• Select the SR-IOV compliant PCIe NIC listed in the “Introduction

• SR-IOV-compliant NICs” section.

• Have two GBIC modules and fibre cables for testing the SR-IOV virtual function.

Figure 2: Illustration of PCIe NIC, GBIC modules, and fibre cables

Technical white paper | Microsoft Windows Server 2012 SR-IOV

6

Configure System

After your server is prepared, you are ready to install the NIC. For information about the supported HP SR-IOV NICs, see the “

SR-IOV-compliant NICs” section.

1. Complete the following steps to install the SR-IOV-compliant PCIe NIC:

A. Install SR-IOV stand up NIC into one of the PCIe slots.

B. Insert the two GBIC modules into the ports.

C. Connect the fibre cables.

Figure 3: Install the PCIe NIC, GBIC modules, and connect fibre cables in this manner to test VMs

Important

As an example, Figure 3 illustrates the cable connections necessary for testing the SR-IOV and VM configuration connections with a ProLiant DL580 G7 server. The connections for your server may vary slightly. The cables will remain connected like this for the remainder of the procedures in this paper. After completing the steps in the “Making sure that the VM is enabled” section, connect the fibre cables to your switch as necessary for your network environment.

2. Make sure that the peripheral modules (monitor, keyboard, and mouse) and power cable are connected to the server.

3. Power on the server.

4. Load the HP ROM-Based Setup Utility (RBSU) by pressing the F9 key during POST when prompted.

5. Complete the following steps to enable the PCIe NIC and SR-IOV (See Figure 4).

Note that the selection names that display for your server may vary slightly:

A. Select the following options to enable the PCIe NIC: System Options Processor Options: Enable Intel Virt Tech and Intel Vt-D

B. Select the following options to enable SR-IOV: Advanced Options Advanced System ROM Options: Enable SR-IOV

Technical white paper | Microsoft Windows Server 2012 SR-IOV

7

6. Save changes, and then exit RBSU.

Figure 4: Use the RBSU to enable SR-IOV

Update the Intelligent Provisioning firmware

Before installing Windows Server 2012, update the IP firware by following the steps provided in the HP Intelligent Provisioning User Guide, located at hp.com/go/intelligentprovisioning.

Install Windows Server 2012

Install the Windows Server 2012 OS by booting the Windows Server 2012 media into the DVD drive, and then following the installation screens.

For tips on installing Windows Server 2012, see the Microsoft website at microsoft.com/en-us/library/jj134246.aspx.

Installing the HP SPP

The benefit of using HP drivers is that they have been validated and known to be supported in HP server platforms.

You can download the HP SPP from hp.com/go/spp/download or for specific instructions on how to download and install HP SPP 2012.10.0 (or later) follow the steps in the Release Notes, which are available at hp.com/pub/softlib2/software1/doc/p1822529277/v81463/SPP2012.10.0rev1ReleaseNotes.pdf.

When you get to the Source Selection screen, continue with the following steps to deploy the updates:

1. Verify that the directory path in the Directory field has the location of the smart components from the extracted Supplement, and then select Start Inventory. HP Smart Update Manager (HP SUM) performs an inventory of the installed hardware and software, and then checks for available updates.

2. After the inventory and discovery finish, the Select Installation Hosts screen appears. Select either the local host or one (or more) remote hosts for Supplement deployment.

3. After selecting the host(s), the Select bundle filter screen shows the Supplement bundle information. Select the bundle and the appropriate filter options. For remote deployments, additional screens allow you to update information on a per-host basis.

Technical white paper | Microsoft Windows Server 2012 SR-IOV

8

4. After selecting the bundle for all hosts being updated, open the Select Items to be Installed screen to complete the following tasks:

A. Select the components for installation.

B. If necessary, configure the components. The Configure Now link is not available when running Windows Server 2012 with the Server Core option. To configure components:

i. Access the system as a remote host using HP SUM, where HP SUM is running on a system with a supported Windows OS.

ii. Configure the components and resolve any failed dependencies before deploying the OS.

C. Review the revision history of the components.

5. After selecting the components, click Install. When the installation finishes, the Installation Results screen displays. If the Supplement installs successfully, the process is complete. If one or more components did not install successfully, complete the following steps:

A. Exit HP SUM.

B. Make the required update to the environment.

Configure Server Manager

To configure Server Manager, complete the processes in the following sections.

Add “IOVEnableOverride” key to the Windows Registry (on servers with an AMD processor)

If you server has an AMD processor, complete the steps in this section. If your server has an Intel processor, procede to the next section.

To avoid an issue that causes Windows Event Log to display event 12607 “IOV Fails,” add the registry key “IOVEnableOverride” to the Windows Registry by completing the following steps:

1. Launch the Windows PowerShell 3.0 CLI by clicking on the PowerShell icon.

2. Type the following to add the registry key:

reg add HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Virtualization /v

IOVEnableOverride /t REG_DWORD /d 1

See Figure 5 for a sample of a successful addition of Windows Registry key.

3. Restart the server after you add the registry key successfully.

Figure 5: Add “IOVEnableOverride” to the Windows Registry

Add the Hyper-V role

To test the SR-IOV virtual function (VF) and physical function (PF), complete the following steps to add the Hyper-V role and create a test VM. For tips on Installing the Hyper-V Role and Configuring a Virtual Machine, see microsoft.com/en-us/library/hh846766.aspx:

1. Launch the Windows Server 2012 and add the Hyper-V role.

2. Create a test VM.

3. Install a guest OS.

Verifying SR-IOV NIC support

Use the following PowerShell command to make sure that your adapter supports SR-IOV:

Get-NetAdapaterSriov

Technical white paper | Microsoft Windows Server 2012 SR-IOV

9

The Sriov Support field should report Supported as in Figure 6.

Figure 6: Verify NIC support for SR-IOV

If the Get-NetAdapterSriov does not report “Supported” and your NIC should support SR-IOV, check the following:

• Make sure that your server supports SR-IOV.

• Make sure that your server has the latest System ROM and firmware.

If your server supports SR-IOV and the firmware is up-to-date, check the following:

• If you see “Missing ACS,” try plugging in the SR-IOV NIC in a different PCI Express slot. It is possible that a PCI Express Switch on the server does not support SR-IOV.

• If you see “Missing PF Driver,” confirm that your NIC supports Windows 2012 SR-IOV, and then run Windows Update and upgrade your NIC drivers.

• If you see “No bus resources,” try disabling unused devices from the System BIOS settings.

• If you see “No IOMMU support,” verify the following:

– Confirm that the Hyper-V role is enabled.

– Confirm that SR-IOV is enabled in the RBSU. For instructions on enabling SR-IOV, see the Configure System section.

Technical white paper | Microsoft Windows Server 2012 SR-IOV

10

Set up a virtual switch

Complete the following steps to set up a virtual switch in SR-IOV mode using the Virtual Switch Manager:

1. Type a name for the virtual switch. See item A in Figure 6. We use the name “HP SRIOV” in this example.

2. Click on the radion button for External network. See item B in Figure 6.

3. Choose port to which you want to connect to the virtual switch. See item C in Figure 6. In this example, we use the port “HP Ethernet 10 Gb 2-port 560SFP+ Adapter #3.”

4. Select the check box for Enable single root I/O virtualization (SR-IOV). See item D in Figure 7.

5. Click Apply, and then click OK. Refer to picture below for a virtual switch setting in SR-IOV mode.

Figure 7: Configure a virtual switch for SR-IOV mode

Technical white paper | Microsoft Windows Server 2012 SR-IOV

11

Enable SR-IOV on a Virtual Machine

To enable SR-IOV on the VM, complete the following steps:

1. Choose the virtual switch that you created previously. See item A in Figure 8.

2. Under the Hardware Acceleration option, click the check the box to Enable SR-IOV. See item B in Figure 8.

3. Click Apply, and then click OK.

Figure 8: Enable SR-IOV for your VM

4. Verify that there no boxes checked in the Advanced Features. If you clear any check boxes, click Apply, and then click OK to accept the changes.

5. Start the VM by clicking on the Run radio button.

6. Confirm the VM is now in the “running” state.

7. Click the “Summary” tab of the test VM and confirm the Heartbeat indicates “OK (Applications Healthy).” See Figure 9.

Technical white paper | Microsoft Windows Server 2012 SR-IOV

12

Figure 9: Start the VM

Making sure that the VM is enabled

To test the SR-IOV physical functions (PF) and virtual functions (VF), complete the following steps:

1. Launch Windows PowerShell 3.0.

2. To confirm that IOV support and IovVirtualFunction are enabled, type the PowerShell command:

get-vmswitch | fl

3. Verify the VM name.

4. Verify the following parameters. These parameters are set as follows when IOV support and IovVirtualFunction are enabled:

– IovEnabled = True

– IovVirtualFunctionCount = 32

– IovVirtualFunctionInUse = 1

– SriovSupport = Supported

– NumVFs = 62

Technical white paper | Microsoft Windows Server 2012 SR-IOV

13

To confirm that the VM is active in Hyper-V Manager, click the Networking tab of the test VM and confirm the Status indicates OK (SR-IOV active). See Figure 10.

Figure 10: Confirm the VM’s PF and VF functionality

5. Connect the fibre cables to the switch as necessary for your network environment.

Recommendations for guest OS configurations

We recommend the following guest OS configuration guidelines:

• Use a 64-bit guest OS to ensure that it can address up to 2 GB of virtual address space per process.

• Make sure that the server has adequate RAM for all of the VMs. You can calculate the RAM necessary for optimal Hyper-V performance as follows:

Minimum RAM in MB = 300+512+(VDIMM+32*1+8*(VDIMM-1024)/1024)*NumVFs

Note

VDIMM is the amount of virtual memory that you want each VM to use. NumVFs is the number of VFs that have been initialized for IOVSwitch.

• For optimal performance with processor intensive applications, configure a 1:1 processor ratio for virtual processors:logical processors. Make sure that you have sufficient CPU logical cores.

Troubleshooting the VM enablement issues

To troubleshoot issues, start by checking the following:

1. Confirm that the RBSU parameters are set up correctly.

2. Ensure the fibre cable is connected to the GBIC modules when testing SR-IOV with the VM.

Check the IovVirtual FunctionCount and IovVirtualFunctionInUse, type the PowerShell command:

get-vmswitch | fl

Technical white paper | Microsoft Windows Server 2012 SR-IOV

14

The messages in the IovSupportReasons may indicate the cause of the issue (see Figure 11).

Figure 11: Check for SR-IOV support issues

If IovSupportReasons states that the PCIe hardware does not support Access Control Services (ACS) at any root port as indicated in Figure 12, confirm that the Windows Registry includes the IOVEnableOverride key.

Figure 12: IOVEnableOvrride key is missing from Windows Registry

Use the following PowerShell commandlets to help with troubleshooting SR-IOV enablement issues:

• To check if IovSupport is enabled, type:

get-vmhost | fl

• To verify that SR-IOV is supported, and the number of VFs, type:

get-netadapterSRIOV | fl

• To see why the SR-IOV status is degraded, type:

get-vmnetworkadapter | fl

Known issues and workarounds

This section outlines a few potential issues that you may encounter with the Windows Server 2012 OS, SR-IOV feature enablement, and ProLiant servers. We are actively working to resolve all the issues.

Table 3. Known issues and workarounds

Issues

IovSupport is not enabled

Description The PCIe hardware does not support Access Control Services (ACS) at any root port.

Workaround/Solution Add the registry key - IOVEnableOverride. For instructions on adding the registry key, see the “Add “IOVEnableOverride” key to the Windows Registry” section.

Cannot create a virtual switch using PowerShell3.0 commandlets

Description Although the link status for an HP Ethernet 10Gb 2-port 560SFP+ Adapter shows running, a message displays that the adapter is not enabled or not functioning. The message displays when creating a SRIOV-enabled Virtual Switch Powershell.

Workaround/Solution We are working with Microsoft and Intel to resolve this issue.

Technical white paper | Microsoft Windows Server 2012 SR-IOV

15

Table 3. Known issues and workarounds

Issues

Certain ProLiant servers do not support ACS and ARI functions in all PCIe slots.

Description The PCIe slots noted for the following servers do not support ACS and ARI functions. If you need advanced SR-IOV functionality, do not install the PCIe NIC to be used with SR-IOV in the following slots:

• HP ProLiant DL360e Gen8 Server: Slot 2

• HP ProLiant DL380e Gen8 Server: Slot 4

• HP ProLiant DL380p Gen8 Server: Slot 3

• HP ProLiant DL560 Gen8 Server: Slot 3

• HP ProLiant ML350e Gen8 Server: Slot 5 and Slot 6

• HP ProLiant ML350p Gen8 Server: Slot 5

Workaround/Solution We are working to resolve this issue. We will be documenting information about this issue in a Customer Advisory, which will be available at hp.com/portal/site/hpsc/public/kb/docDisplay/?docId=emr_na-c03517970.

May not be able to create the maximum number of VFs on ProLiant DL980 G7 server

Description A ProLiant DL980 G7 server with four SR-IOV enabled NICs may create a maximum of 256 VFs.

Workaround/Solution We are working with Microsoft and Intel to resolve this issue.

ProLiant blade servers with a Flex-10 10Gb 2-port 530FLB adapter indicate that the NIC does not support SR-IOV.

Description When making sure that a VM is enabled on a Flex-10 10Gb 2-port 530FLB adapter in a ProLiant blade server, the results indicate that the NIC does not support SR-IOV.

Workaround/Solution The Flex-10 10Gb 2-port 530FLB adapter does support Windows Server 2012 SR-IOV.

To resolve this issue, complete the following steps:

1. Update the drivers using the latest SPP.

2. Restart the server, and press ctrl+s during POST to view the Comprehensive Configuration Menu (CCM).

3. Change the Mode setting for the Flex-10 10Gb 2-port 530FLB adapter from “multi-function” to “single function.”

Sign up for updates

hp.com/go/getupdated

Share with colleagues

Rate this document

© Copyright 2012, 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only

warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should

be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Intel, Intel Itanium, and Intel Xeon are trademarks of Intel Corporation in the

United States and other countries. AMD is a trademark of Advanced Micro Devices, Inc.

TC1302955, Created October 2012, Updated April 2013

Resources, contacts, or additional links

HP Windows Server 2012 Home Page hp.com/go/ws2012

Microsoft Windows Server 2012 Home Page microsoft.com/en-us/server-cloud/windows-server/default.aspx

HP Ethernet 10Gb 2-port 560FLB Adapter hp.com/products/servers/networking/560FLB/index.html

HP Ethernet 10Gb 2-port 560SFP+ Adapter hp.com/products/servers/networking/560SFP/index.html

ProLiant technology papers hp.comhp.com/servers/technology