Click here to verify the latest version of this document. Created October 2012.

Technical white paper

Implementing Windows Server 2012 SR-IOV on HP ProLiant Servers

Table of contents

Introduction 2

Overview of SR-IOV 2

Windows Server 2012 recommended system configurations 3

SR-IOV-compliant ProLiant servers 3

SR-IOV-compliant NICs 4

Enabling the SR-IOV feature 5 Prepare Materials 5 Configure System 6 Install Windows Server 2012 7 Installing the HP SPP 7 Installing the HP SR-IOV enabled NIC driver 8 Configure Server Manager 8 Enable SR-IOV on a Virtual Machine 10

Making sure that the VM is enabled 11

Recommendations for guest OS configurations 12

Troubleshooting the VM enablement issues 13

Known issues and workarounds 14

For more information 16

Call to action 16

2

Introduction Microsoft Windows Server 2012 supports Single Root I/O Virtualization (SR-IOV) as a new feature for Hyper-V. SR-IOV is included in a Peripheral Component Interconnect Special Interest Group (PCI-SIG) specification for partitioning PCI Express (PCIe) adapter bandwidth. Windows Server 2012 supports SR-IOV through SR-IOV capable PCIe network adapters (NICs).

Partitioning port bandwidth provides several benefits, including:

• Increased I/O data throughput

• Reduced CPU utilization for processing network traffic

• Reduced network path latency

• Support for live migration

For more information on the SR-IOV standard and industry support for the standard, go to the PCI-SIG site at: http://www.pcisig.com.

This paper focuses on implementing SR-IOV within Windows Server 2012. The paper includes the requirements and recommendations for using the Windows Server 2012 SR-IOV feature on HP ProLiant Servers.

Before implementing SR-IOV, first make sure that your server meets the Windows Server 2012 configuration requirements as established by Microsoft. For details on these requirements, see the Microsoft website at http://technet.microsoft.com/library/jj134246.aspx.

For detailed information or to download Windows Server 2012, see the Microsoft Windows Server 2012 home page at http://www.microsoft.com/en-us/server-cloud/windows-server/default.aspx.

Overview of SR-IOV SR-IOV allows multiple VMs to share a single SR-IOV-capable PCIe NIC while retaining the performance benefit of one PCIe device to one VM association.

The SR-IOV standard builds on the following PCI standards to extend the PCI configuration space and define access to Virtual Functions:

• PCIe

• Alternative Routing ID (ARI)

• Address Translation Services (ATS)

• Function Level Reset (FLR)

By assigning a Virtual Function (VF) to each VM, multiple VMs can share a single SR-IOV capable PCIe NIC that may have just one physical network port.

A Virtual Function is a function that has an association with a Physical Function. Each partition of the PCIe NIC appears in the PCI configuration space as a Virtual Function with its own configuration space and Base Address Registers (BAR). It shares the bandwidth of the PCIe NIC with other Physical Functions or Virtual Functions. Characteristics of a Virtual Function include:

• Support for native I/O virtualization

• A PCIe configuration space that defines a unique set of Message Signaled Interrupts (MSI) and MSI-eXtended (MSI-X) vectors

3

A Physical Function is a PCIe function that supports the SR-IOV capabilities. It is a standard PCIe device with its own full PCI configuration space and set of resources to manage Virtual Functions. Characteristics of a Physical Function include:

• SR-IOV Extended Capability as part of its configuration space

• Configuration information for the Virtual Functions associated with it

• Base Address Registers (BAR) configuration and Virtual Function type for any Virtual Functions associated with it

SR-IOV implements bandwidth partitioning by allocating multiple Virtual Functions to a Physical Function. The Virtual Functions appear as PCI devices that are part of the physical PCI device by their resources (queues and register sets). Instead of using the Virtual Machine Manager for VM processing, SR-IOV bypasses the VMbus with dedicated hardware queues and a round-robin scheduler to reduce processor overhead.

Because SR-IOV enumerates Virtual Functions in a hardware-based PCI configuration space, it is well suited for providing hardware support for virtualization of fixed-function devices. A Virtual Function and Physical Function that are associated must be of the same type because the Virtual Function inherits its Vendor ID and Device ID from its associated Physical Function’s configuration space fields.

Windows Server 2012 recommended system configurations The PCI subsystem and the Windows Server 2012 kernel contain the core SR-IOV implementation, so both must support SR-IOV. In addition, the PCIe NIC driver must support Virtual Functions and Physical Functions.

Processor virtualization (such as AMD-v and Intel VT-x) and chipset virtualization (such as AMD-Vi and Intel VT-d) allow a VM and guest OS to share a SR-IOV-capable PCIe NIC. This allows the Virtual Machine Manager to assign one or more Virtual Functions to each VM.

In addition to the Windows Server 2012 configuration requirements, your server configuration must also meet the following criteria to use SR-IOV:

• The processor must support Second Level Address Translation (SLAT) and I/O Memory Management Unit (IOMMU) technology. To meet these requirements, we recommend using either of the following processors. See the “SR-IOV-compliant ProLiant servers” section for details on our servers that include these processors and support Windows Server 2012 SR-IOV:

– Intel Xeon Processors with Intel64 architecture with VT-x and EPT

– Advanced Micro Devices, Inc. (AMD64) Opteron Series Processors with AMD-Vi and NPT

• The System ROM must be capable of enabling and describing the SR-IOV capabilities to the Windows Server 2012 operating system.

• You must use an SR-IOV–capable NIC and driver.

SR-IOV-compliant ProLiant servers Table 1 lists the ProLiant servers and minimum System ROM version that are required for Windows Server 2012 and the SR-IOV feature.

IMPORTANT: SR-IOV requires both hardware and System ROM compliance to work. Your server must have the minimum System ROM version listed in the table to enable SR-IOV. If necessary, update the System ROM before beginning enabling SR-IOV.

You can download System ROM updates from the HP Service Pack for ProLiant (HP SPP), which is available at www.hp.com/go/spp/download or the HP Support Center at www.hp.com/go/support.

4

Table 1. Required System ROM versions

Server platform System ROM family System ROM Version

ProLiant BL servers

BL420c Gen8 I30 8/20/2012 or later

BL460c Gen8 I31 8/20/2012 or later

BL465c Gen8 A26 8/20/2012 or later

BL660c Gen8 I32 8/20/2012 or later

ProLiant DL servers

DL360e Gen8 P73 8/20/2012 or later

DL360p Gen8 P71 8/20/2012 or later

DL380e Gen8 P73 8/20/2012 or later

DL380p Gen8 P70 8/20/2012 or later

DL385p Gen8 A28 8/14/2012 or later

DL560 Gen8 P77 8/20/2012 or later

DL580 G7 P65 8/4/2012 or later

DL585 G7 A16 8/14/2012 or later

DL980 G7 P66 7/30/2012 or later

ProLiant ML servers

ML350e Gen8 J02 8/20/2012 or later

ML350p Gen8 P72 8/20/2012 or later

ProLiant Blade/SL line servers

SL230s Gen8 P75 8/20/2012 or later

SL250s Gen8 P75 8/20/2012 or later

SL270s Gen8 P75 8/20/2012 or later

SR-IOV-compliant NICs At this time, you must use either one of the following NICs to implement SR-IOV:

• HP Ethernet 10Gb 2-port 560FLB Adapter: This is a dual port 10Gb Ethernet FlexibleLOM mezzanine adapter for the SR-IOV-compliant ProLiant BL Gen8 servers. For details on this adapter, see the HP website at http://h18004.www1.hp.com/products/servers/networking/560FLB/index.html.

• HP Ethernet 10Gb 2-port 560SFP+ Adapter: This is a small form-factor pluggable (SFP+) adapter that you can insert into an SR-IOV-compliant ProLiant DL, ML, or SL G7 or Gen8 server. For details on this adapter, see the HP website at http://h18004.www1.hp.com/products/servers/networking/560SFP/index.html.

5

The drivers for these NICs are available on the Windows Server 2012 media or the HP SPP, which is available at www.hp.com/go/spp.

We will update this document to include newly supported configurations as our testing progresses.

Enabling the SR-IOV feature Figure 1 provides an overview of the process for enabling the SR-IOV feature.

Figure 1: SR-IOV enablement process overview

Prepare Materials

As preparation for enabling the SR-IOV feature, make sure that your server and components are supported with Windows Server 2012 and SR-IOV:

• Select a server from the recommended system platforms listed in Table 1 of the “SR-IOV-compliant ProLiant servers” section.

• Make sure the the ProLiant server has the System ROM version indicated in Table 1 of the “SR-IOV-compliant ProLiant servers” section.

• Make sure that the server has a DVD drive (either installed or attached to the server receiving the installation). If you will be using iLO virtual media with either a HP ProLiant DL or ML server, you will need a license key.

• Select the SR-IOV compliant PCIe NIC listed in the “SR-IOV-compliant NICs” section.

Prep Materials

•SR-IOV-compliant ProLiant Server with the required ROM version •SR-IOV-compliant NIC , two GBIC modules, and fibre cables to test VMs •Windows Server 2012 OS DVD

Configure System

• Install SR-IOV-compliant NIC •Use the ROM Based Setup Utility (RBSU) to enable SR-IOV • Install the Windows Server 2012 OS • Install the HP SPP • Install the HP SR-IOV driver

Configure Server

Manager

•Use PowerShell to add the registry key •Add the Hyper-V role •Create VMs • Install the guest OS •Configure the virtual switch settings

Configure Virtual

Machiine

•Configure NIC settings •Confirm the Advanced Feature setting •Start the VM

Test PF, VF Functions and Troubleshoot

•Confirm SR-IOV support is enabled •Verify the number of VMs •Confirm IovVirtualFunctionCount and IovVirtualFunctionsInUse •Confirm SR-IOV status is OK •Troubleshoot system issues as required

6

• Have two GBIC modules and fibre cables for testing the SR-IOV virtual function.

Figure 2: Illustration of PCIe NIC, GBIC modules, and fibre cables

Configure System

After your server is prepared, you are ready to install the NIC. For information about the supported HP SR-IOV NICs, see the “SR-IOV-compliant NICs” section. 1. Complete the following steps to install the SR-IOV-compliant PCIe NIC:

a. Install SR-IOV stand up NIC into one of the PCIe slots.

b. Insert the two GBIC modules into the ports.

c. Connect the fibre cables.

Figure 3: Install the PCIe NIC, GBIC modules, and connect fibre cables in this manner to test VMs

IMPORTANT: As an example, Figure 3 illustrates the cable connections necessary for testing the SR-IOV and VM configuration connections with a ProLiant DL580 G7 server. The connections for your server may vary slightly. The cables will remain connected like this for the remainder of the procedures in this paper. After completing the steps in the “Making sure that the VM is enabled” section, connect the fibre cables to your switch as necessary for your network environment.

7

2. Make sure that the peripheral modules (monitor, keyboard, and mouse) and power cable are connected to the server.

3. Power on the server.

4. Load the HP ROM-Based Setup Utility (RBSU) by pressing the F9 key during POST when prompted.

5. Complete the following steps to enable the PCIe NIC and SR-IOV (See Figure 4).

Note that the selection names that display for your server may vary slightly:

a. Select the following options to enable the PCIe NIC:

System Options Processor Options: Enable Intel® Virt Tech and Intel® Vt-D

b. Select the following options to enable SR-IOV:

Advanced Options Advanced System ROM Options: Enable SR-IOV

6. Save changes, and then exit RBSU.

Figure 4: Use the RBSU to enable SR-IOV

Install Windows Server 2012

Install the Windows Server 2012 OS by booting the Windows Server 2012 media into the DVD drive, and then following the installation screens.

For tips on installing Windows Server 2012, see the Microsoft website at http://technet.microsoft.com/en-us/library/jj134246.aspx.

Installing the HP SPP

The benefit of using HP drivers is that they have been validated and known to be supported in HP server platforms.

You can download the HP SPP from www.hp.com/go/spp/download or for specific instructions on how to download and install HP SPP 2012.10.0 (or later) follow the steps in the Release Notes, which are available at http://ftp.hp.com/pub/softlib2/software1/doc/p1822529277/v81463/SPP2012.10.0rev1ReleaseNotes.pdf.

When you get to the Source Selection screen, continue with the following steps to deploy the updates:

1. Verify that the directory path in the Directory field has the location of the smart components from the extracted Supplement, and then select Start Inventory.

8

HP Smart Update Manager (HP SUM) performs an inventory of the installed hardware and software, and then checks for available updates.

2. After the inventory and discovery finish, the Select Installation Hosts screen appears. Select either the local host or one (or more) remote hosts for Supplement deployment.

3. After selecting the host(s), the Select bundle filter screen shows the Supplement bundle information. Select the bundle and the appropriate filter options. For remote deployments, additional screens allow you to update information on a per-host basis.

4. After selecting the bundle for all hosts being updated, open the Select Items to be Installed screen to complete the following tasks:

a. Select the components for installation.

b. If necessary, configure the components. The Configure Now link is not available when running Windows Server 2012 with the Server Core option. To configure components:

i. Access the system as a remote host using HP SUM, where HP SUM is running on a system with a supported Windows OS.

ii. Configure the components and resolve any failed dependencies before deploying the OS.

c. Review the revision history of the components.

5. After selecting the components, click Install. When the installation finishes, the Installation Results screen displays. If the Supplement installs successfully, the process is complete. If one or more components did not install successfully, complete the following steps:

a. Exit HP SUM.

b. Make the required update to the environment.

Installing the HP SR-IOV enabled NIC driver

Install the HP SR-IOV enabled NIC driver by completing the following steps:

1. Copy the component package cp017638 from the HP SPP to your server Windows desktop.

2. Double click on the cp017638 icon on your server Windows desktop to launch the component package cp017638 setup.

3. Choose Install from the component package cp017638 setup.

4. Click Install again when the component package cp017638 setup screen appears.

The message “The installation procedure was completed successfully” will display upon completion of the HP SR-IOV driver.

Configure Server Manager

To configure Server Manager, complete the processes in the following sections.

Add “IOVEnableOverride” key to the Windows Registry To avoid an issue that causes Windows Event Log to display event 12607 “IOV Fails,” add the registry key “IOVEnableOverride” to the Windows Registry by completing the following steps:

1. Launch the Windows PowerShell 3.0 CLI by clicking on the PowerShell icon.

2. Type the following to add the registry key:

reg add HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Virtualization /v IOVEnableOverride /t REG_DWORD /d 1

See Figure 5 for a sample of a successful addition of Windows Registry key.

3. Restart the server after you add the registry key successfully.

9

Figure 5: Add “IOVEnableOverride” to the Windows Registry

Add the Hyper-V role To test the SR-IOV virtual function (VF) and physical function (PF), complete the following steps to add the Hyper-V role and create a test VM. For tips on Installing the Hyper-V Role and Configuring a Virtual Machine, see http://technet.microsoft.com/en-us/library/hh846766.aspx:

1. Launch the Windows Server 2012 and add the Hyper-V role.

2. Create a test VM.

3. Install a guest OS.

Set up a virtual switch Complete the following steps to set up a virtual switch in SR-IOV mode using the Virtual Switch Manager:

1. Type a name for the virtual switch. See item A in Figure 6. We use the name “HP SRIOV” in this example.

2. Click on the radion button for External network. See item B in Figure 6.

3. Choose port to which you want to connect to the virtual switch. See item C in Figure 6. In this example, we use the port “HP Ethernet 10 Gb 2-port 560SFP+ Adapter #3”.

4. Select the check box for Enable single root I/O virtualization (SR-IOV). See item D in Figure 6.

5. Click Apply, and then click OK. Refer to picture below for a virtual switch setting in SR-IOV mode.

Figure 6: Configure a virtual switch for SR-IOV mode

10

Enable SR-IOV on a Virtual Machine

To enable SR-IOV on the VM, complete the following steps:

1. Choose the virtual switch that you created previously. See item A in Figure 7.

2. Under the Hardware Acceleration option, click the check the box to Enable SR-IOV. See item B in Figure 7.

3. Click Apply, and then click OK.

Figure 7: Enable SR-IOV for your VM

4. Verify that there no boxes checked in the Advanced Features. If you clear any check boxes, click Apply, and then click OK to accept the changes.

5. Start the VM by clicking on the Run radio button.

6. Confirm the VM is now in the “running” state.

11

7. Click the “Summary” tab of the test VM and confirm the Heartbeat indicates “OK (Applications Healthy) . See Figure 8.

Figure 8: Start the VM

Making sure that the VM is enabled To test the SR-IOV physical functions (PF) and virtual functions (VF), complete the following steps:

1. Launch Windows PowerShell 3.0.

2. To confirm that IOV support and IovVirtualFunction are enabled, type the PowerShell command:

get-vmswitch | fl

3. Verify the VM name.

4. Verify the following parameters. These parameters are set as follows when IOV support and IovVirtualFunction are enabled:

– IovEnabled = True

– IovVirtualFunctionCount = 32

– IovVirtualFunctionInUse = 1

– SriovSupport = Supported

– NumVFs = 62

12

To confirm that the VM is active in Hyper-V Manager, click the Networking tab of the test VM and confirm the Status indicates OK (SR-IOV active). See Figure 9.

Figure 9: Confirm the VM’s PF and VF functionality

5. Connect the fibre cables to the switch as necessary for your network environment.

Recommendations for guest OS configurations We recommend the following guest OS configuration guidelines:

• Use a 64-bit guest OS to ensure that it can address up to 2 GB of virtual address space per process.

• Make sure that the server has adequate RAM for all of the VMs. You can calculate the RAM necessary for optimal Hyper-V performance as follows:

VDIMM is the amount of virtual memory that you want each VM to use.

NumVFs is the number of VFs that have been initialized for IOVSwitch.

Minimum RAM in MB = 300+512+(VDIMM+32*1+8*(VDIMM-1024)/1024)*NumVFs

• For optimal performance with processor intensive applications, configure a 1:1 processor ratio for virtual processors:logical processors. Make sure that you have sufficient CPU logical cores.

13

Troubleshooting the VM enablement issues To troubleshoot issues, start by checking the following:

1. Confirm that the RBSU parameters are set up correctly.

2. Ensure the fibre cable is connected to the GBIC modules when testing SR-IOV with the VM.

To check the IovVirtual FunctionCount and IovVirtualFunctionInUse, type the PowerShell command:

get-vmswitch | fl

The messages in the IovSupportReasons may indicate the cause of the issue (see Figure 10).

Figure 10: Check for SR-IOV support issues

If IovSupportReasons states that the PCIe hardware does not support Access Control Services (ACS) at any root port as indicated in Figure 11, confirm that the Windows Registry includes the IOVEnableOverride key.

Figure 11: IOVEnableOvrride key is missing from Windows Registry

The following PowerShell commandlets may also help with troubleshooting SR-IOV enablement issues:

• To check if IovSupport is enabled, type:

get-vmhost | fl

• To check if SRIOV is supported, and the number of VFs, type:

get-netadapterSRIOV | fl

• To check reasons for SRIOV status: Degraded, type:

get-vmnetworkadapter | fl

14

Known issues and workarounds This section outlines a few potential issues that you may encounter with the Windows Server 2012 OS, SR-IOV feature enablement, and ProLiant servers. We are actively working to resolve all the issues.

Issues

IovSupport is not enabled

Description The PCIe hardware does not support Access Control Services (ACS) at any root port.

Workaround/Solution

Add the registry key - IOVEnableOverride. For instructions on adding the registry key, see the “Add IOVEnableOverride key to the Windows Registry” section.

Cannot create a virtual switch using PowerShell3.0 commandlets

Description Although the link status for an HP Ethernet 10Gb 2-port 560SFP+ Adapter shows running, a message displays that the adapter is not enabled or not functioning. The message displays when creating a SRIOV-enabled Virtual Switch Powershell.

Workaround/Solution

We are working with Microsoft and Intel to resolve this issue.

Certain ProLiant servers do not support ACS and ARI functions in all PCIe slots.

Description The PCIe slots noted for the following servers do not support ACS and ARI functions. If you need advanced SR-IOV functionality, do not install the PCIe NIC to be used with SR-IOV in the following slots:

• HP ProLiant DL360e Gen8 Server: Slot 2

• HP ProLiant DL360p Gen8 Server: Slot 3

• HP ProLiant DL380e Gen8 Server: Slot 4

• HP ProLiant DL380p Gen8 Server: Slot 3

• HP ProLiant DL560 Gen8 Server: Slot 3

• HP ProLiant ML350e Gen8 Server: Slot 5 and Slot 6

• HP ProLiant ML350p Gen8 Server: Slot 5

Workaround/Solution

We are working to resolve this issue. We will be documenting information about this issue in a Customer Advisory, which will be available at http://h20564.www2.hp.com/portal/site/hpsc/public/kb/docDisplay/?docId=emr_na-c03517970.

HP SUM does not install the HP SR-IOV enabled NIC driver

Description HP SUM does not install the HP SR-IOV enabled NIC driver or update the Windows Server 2012 in-box SR-IOV enabled NIC driver.

Workaround/Solution

Install the HP SR-IOV enabled NIC driver by completing the following steps:

1. Copy the component package cp017638 from the HP SPP to your server Windows desktop.

2. Double click on the cp017638 icon on your server Windows desktop to launch the component package cp017638 setup.

3. Choose Install from the component package cp017638 setup.

4. Click Install again when the component package cp017638 setup screen appears.

The message “The installation procedure was completed successfully” will display upon completion of the HP SR-IOV driver.

15

Issues

May not be able to create the maximum number of VFs on ProLiant DL980 G7 server

Description A ProLiant DL980 G7 server with four SR-IOV enabled NICs may create a maximum of 256 VFs.

Workaround/Solution

We are working with Microsoft and Intel to resolve this issue.

16

For more information

For additional information, refer to the resources listed below.

Resource description Web address

HP Windows Server 2012 Home Page

Microsoft Windows Server 2012 Home Page

www.hp.com/go/ws2012

ProLiant technology papers

www.microsoft.com/en-us/server-cloud/windows-server/default.aspx

www.hp.com/servers/technology

HP Ethernet 10Gb 2-port 560FLB Adapter http://h18000.www1.hp.com/products/servers/networking/560FLB/index.html

HP Ethernet 10Gb 2-port 560SFP+ Adapter http://h18000.www1.hp.com/products/servers/networking/560SFP/index.html

Call to action

Send comments about this paper to TechCom@HP.com.

Follow us on Twitter: http://twitter.com/ISSGeekatHP.

Get connected hp.com/go/getconnected

Current HP driver, support, and security alerts delivered directly to your desktop

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Intel, Intel Itanium, and Intel Xeon are trademarks of Intel Corporation in the United States and other countries. AMD is a trademark of Advanced Micro Devices, Inc.

TC1210936, Created October 2012