WHI TE PA P ER

Virtual Network Exceleration

OCe14000 Ethernet Network

Adapters High Performance Networking for Enterprise

Virtualization and the Cloud

2 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Emulex OCe14000 Ethernet Network Adapters

High Performance Networking for Enterprise Virtualization and the Cloud

Abstract

The OCe14000 family of Ethernet Network Adapters is the latest and most advanced generation of Emulex Ethernet adapters. Powered by the XE100 controller, OCe14000 adapters offer enhanced support for virtualization, cloud and hyperscale cluster deployments. This paper highlights support for Emulex’s Virtual Network eXceleration initiative that delivers support for Virtual Network Fabrics, Single Root I/O Virtualization, Universal Multi-Channel™ and NIC teaming amongst its many capabilities.

Table of Contents

Emulex OneConnect® adapters powered by the XE100 Ethernet controller ............................................................................... 4

Virtual Network Exceleration™ (VNeX) ........................................................................................................................................... 5

Scaling Virtualization Host I/O Performance .................................................................................................................................. 6

Windows Server Dynamic VMQ and VMware NetQueue ........................................................................................................... 6

Receive Segment Coalescing /Large Receive Offload ................................................................................................................ 6

Receive Side Scaling .................................................................................................................................................................... 7

Other Offloads ............................................................................................................................................................................. 7

Scaling a Multi-tenant Cloud Infrastructure ................................................................................................................................... 7

Virtual Network Fabrics - Overcoming Limitations of 802.1Q VLAN Based Network Isolation ................................................ 7

Virtual Network Fabrics – Delivering Virtual Machine Mobility ................................................................................................. 8

Virtual Network Fabrics – The Path to Software Defined Networking .................................................................................... 10

Emulex Enhances VNF Performance .............................................................................................................................................. 11

Optimizing Adapter I/O Bandwidth................................................................................................................................................ 11

Single Root I/O Virtualization ..................................................................................................................................................... 11

Universal Multi-Channel and Assigned Bandwidth ................................................................................................................... 12

Continuous Networking Availability .............................................................................................................................................. 13

Conclusion ....................................................................................................................................................................................... 14

3 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Networking – The common foundation

Networking is an essential infrastructure element of today’s data center applications and services. Networking utilization has skyrocketed as the industry has transitioned from the client-server computing paradigm to web-based applications delivery, pervasive virtualization and emergent cloud architectures.

Driven by this paradigm shift, Cisco’s Global Cloud Index1 forecast suggests some fundamental outcomes in networking traffic patterns by 2016:

1. Total network traffic will grow from 1.8 to 6.6 zettabytes/year.

2. 76% of this traffic will stay within the data center.

1 Cisco Global Cloud Index

1.8 2.6

3.3 4.1

5.2

6.6

0.00

2.00

4.00

6.00

8.00

2011 2012 2013 2014 2015 2016

Zett

abyt

es/

year

Global Data Center IP Traffic Growth

76%

7%

17%

Traffic Destination Forecast - 2016

Within Data Center

Inter Data Center

Data Center to User

4 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

3. 62% of traffic will be in cloud architected data centers, up from 30% in 2011

The IT trends driving these data center network traffic patterns include:

Growth in virtualization-generated, server to server, intra data center traffic such as vMotion, backup, replication and Virtual Desktop Infrastructure (VDI)

Growth of multi-tiered application workloads that increase traffic between database and application servers

Adoption of private, public or hybrid cloud infrastructure and software as a service (SaaS)

Increased consumption of internet streaming video and audio services

The technology powering these fundamental transformations in data center computing is 10Gb Ethernet (10GbE). With the launch of its next generation OneConnect® adapters, Emulex is advancing the state of the art in 10GbE connectivity to enable optimized growth of virtualization and cloud computing.

Emulex OneConnect® adapters powered by the XE100 Ethernet controller Emulex is a leader in high-performance network connectivity, including converged adapter options and enhanced

capabilities for advancing server virtualization that enable a cloud architected infrastructure. The OCe14000 family of 10Gb

(10GbE) and 40Gb (40GbE) Ethernet Network Adapters powered by the XE100 controller, is the latest and most advanced

server connectivity platform from Emulex. Supporting multiple network and storage protocols with powerful and efficient

hardware offloads, OCe14000 adapters have been architected for the dynamic connectivity requirements of today and

future-proofed for the increased demands that tomorrow will bring.

New adapters include:

OCe14101 single-port 10Gb Ethernet Network Adapter

OCe14102 dual-port 10Gb Ethernet Network Adapter

OCe14401 single-port 40Gb Ethernet Network Adapter

30% 39% 46% 52% 57% 62%

70% 61% 54% 48% 43% 38%

2011 2012 2013 2014 2015 2016

Workload Shift

Cloud Data Center Traditional Data Center

5 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Figure 4 – Single-port OCe14101 and Dual-port OCe14102 10GbE Adapter Single-port OCe14401 40GbE Adapter

The OCe14000 family will deliver the industry’s most comprehensive set of features for enterprise cloud and software

defined networking (SDN) deployments. These include:

Greater bandwidth capacity with 40GbE adapters

Hardware offloads for SDN deployments of Virtual Network Fabrics (VNFs) including NVGRE and VXLAN

Enterprise hardened and tested iSCSI and Fibre Channel over Ethernet (FCoE) storage protocol offloads with superior performance, delivering well over one million I/O operations per second (IOPS)

Universal Multi-Channel (UMC) NIC partitioning to optimize bandwidth allocation between workload I/O and virtualization services

Optimization of high-density virtualized servers with Single Root I/O Virtualization (SR-IOV)

Driver capabilities for leveraging multi-core processors to maximize VM density

Working with OEM partners to ensure interoperability and broad availability, 10GbE technology developed by Emulex is

available as LAN on Motherboard (LOM) controllers and as PCI Express (PCIe) adapters for blade and rack-mount servers.

This white paper focuses on the innovations that deliver performance improvements for networks in highly virtualized and

emerging cloud architected data centers.

Virtual Network Exceleration™ (VNeX)

The maturation of virtualization deployments has shifted IT management focus to further leverage this technology by:

Optimizing server hardware investments by increasing VM density – the number of VMs per host server

Virtualizing Tier 1 workloads which have previously been deployed on dedicated server hardware

Adopting cloud services architectures in private or hybrid deployments

The paradigm shift from a purely virtualization driven data center to an infrastructure architecture for supporting cloud

computing is also shifting the focus from pure applications consolidation to tenant consolidation. A tenant can be a

department or a line of business (LoB) in a private cloud or a distinct corporate entity in a hybrid cloud. The networking

implications of this shift are discussed further in this paper.

Aligning with, and complementing these IT trends, is Virtual Network eXceleration (VNeX) – an Emulex technology initiative

that is focused on fully leveraging the benefits of several virtualization and cloud related I/O technologies. The Emulex

VNeX suite delivers multiple benefits including:

6 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Scaling I/O performance in virtualization hosts to support increasing VM density

Scaling networking infrastructure and streamlining deployment for multi-tenant cloud infrastructures

Optimizing adapter I/O bandwidth utilization

Ensuring continuous availability of networking resources

Scaling Virtualization Host I/O Performance Performance enhancements in VM compute capabilities combined with increases in host VM density require advanced NIC

capabilities to ensure the efficient and effective scaling of networking performance and prevent networking I/O

bottlenecks. Windows Server® and VMware vSphere® have both introduced several features to enhance networking

scalability over several platform generations.

The Emulex OCe14000 adapter family supports these scalability features, which ensures maximum I/O performance for

increasingly demanding deployments. These include:

Windows Server Dynamic VMQ and VMware NetQueue Virtual Machine Queue (VMQ), originally introduced in Windows Server 2008 R2, allowed a VMQ-enabled network adapter

to copy received data directly to a VM’s accessible memory. This avoided multiple packet copies and unnecessary virtual

switch routing lookups.

Without VMQ, a single processor in a multi-core CPU host runs all network traffic. With VMQ, interrupts are spread across

multiple processors and a single physical NIC appears as multiple NICs to the VMs, creating the illusion of a dedicated NIC

for each VM. This delivers an overall performance increase for I/O operations and allows a higher host VM density than

might otherwise be possible.

In Windows Server 2012, incoming network traffic processing is dynamically assigned to host processors based on

processor usage and network I/O load, adding and releasing processors for network traffic. This results in improved

network performance, and more importantly, enables the hosting of additional virtual workloads without fear of

networking becoming a choke point for increased VM density.

The OCe14000 family fully supports dynamic VMQ and VMQ capabilities across Windows Server platforms.

VMware NetQueue also allows compatible network adapters to deliver network traffic to the system in multiple receive

queues that can be processed separately, allowing processing to be scaled to multiple CPUs and improving receive-side

networking performance.

When NetQueue is enabled in vSphere, the Emulex OneConnect adapter provides up to eight individual receive queues on

each port and offloads Layer 2 (L2) packet sorting and classification functions from the hypervisor. When Ethernet packets

are routed to a OneConnect network controller, they are placed into specific queues, allowing packet processing to be

scaled to multiple CPUs and significantly improving receive-side networking performance.

Receive Segment Coalescing /Large Receive Offload Receive Segment Coalescing (RSC) is a Windows Server offload technology that helps reduce CPU utilization for receive

side network processing by offloading tasks from the CPU to an RSC-capable network adapter. A NIC enabled for RSC

aggregates multiple TCP/IP packets into a single larger packet (up to 64 kB) before sending it to the host’s network stack

for delivery to applications. RSC improves the receive-side performance of the operating system by reducing the CPU

overhead under network I/O intensive workloads which in turn improves the compute scalability of the server.

7 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Large Receive Offload (LRO) is the equivalent VMware vSphere feature to coalesce TCP packets from the same connection

into a large buffer to reduce CPU interrupts. LRO provides up to a 40% improvement2 in both throughput and CPU

efficiency.

Receive Side Scaling Receive Side Scaling (RSS) enables a NIC to distribute and share the processing of I/O interrupts across multiple processor

cores instead of burdening a single processor core and potentially limiting the NIC’s receive throughput. In Windows

Server 2012, RSS is enhanced to support User Datagram Protocol (UDP) traffic and works on hosts with more than 64

processors3 to achieve highly scalable configurations.

Other Offloads The OCe14000 family additionally supports other offloads including checksum validation for IPv4/IPv6 and TCP payload in

both receive and transmit modes to further unburden the server’s CPUs from network processing tasks.

Scaling a Multi-tenant Cloud Infrastructure Emulex has participated with VMware and Microsoft to deliver solutions for the problem of scaling a cloud infrastructure.

While different in implementation details, these Virtual Networking Fabric (VNF) solutions address two critical

shortcomings of traditional Ethernet networks that limit scalability for cloud architected data centers.

Unique Virtual Local Area Network (VLAN) Identifier IDs limited to 4094 hindering tenant expansion

L2 restrictions for VM communications and VM mobility

Virtual Network Fabrics - Overcoming Limitations of 802.1Q VLAN Based Network Isolation A cloud data center, private or public, supports multiple departments, LoB or individual customers. Each of these entities

requires secure isolation of their individual networks, currently accomplished using 802.1Q VLAN tagging.

VMs belonging to a single group (departmental, LoB, corporate etc.) require isolation based on VLAN tagging for their

network traffic (since VMs communicate at L2). However, the IEEE 802.1Q standard, with its 12-bit namespace for a VLAN

identifier (VID), allows only 4094 VLANs. A top-of-rack switch may connect to dozens of physical servers, each hosting

multiple VMs, each of which may belong to at least one VLAN. A data center can easily contain enough switches to quickly

exceed the VLAN ID limit of 4094.

For a cloud infrastructure, the importance of network scalability is in its ability to securely provision tens of thousands of

isolated tenant networks. VNFs, also known as overlay networks, solve this problem. VNF solutions, from VMware and

Microsoft address this problem through the addition of a new 24-bit identifier to the Ethernet data frame and

encapsulating the frame in an Internet Protocol (IP) packet.

VMware’s VNF solution, known as Virtual Extensible LAN (VXLAN), calls this 24-bit identifier a VXLAN Network Identifier

(VNI) tag, while Microsoft’s Network Virtualization using Generic Routing Encapsulation (NVGRE) named it a Tenant

Network Identifier (TNI) tag.

In either case, this new 24-bit identification tag enables more than 16 million isolated networks to operate within the same

administrative domain, an improvement of many orders of magnitude over the 4,094 VLAN segments discussed above.

2What’s new in VMware vSphere 4 3 Receive Side Scaling

8 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

The figure below shows how the Emulex OCe14000 adds the necessary VNI or TNI tags to support VNF networks. One

advantage of Emulex’s approach is enabling VNF networks without requiring upgrades to the switch infrastructure.

Figure 5 – Emulex encapsulates VXLAN & NVGRE packets

Virtual Network Fabrics – Delivering Virtual Machine Mobility Communications between virtual machines (VMs) as well as the mobility of VMs are restricted to the boundaries of a L2

subnet. This limits the scalability of a cloud infrastructure in multiple usage scenarios.

VM migration due to vMotion

Elastic capacity expansion of applications computing resources to service a tenant may require more servers or

VMs on a different rack or cluster within a different L2 subnet in a different Layer 3 (L3) network. These

additional VMs will need to communicate with the application’s VM on the primary rack or cluster.

Migration of a VM to another server cluster, that is, workload mobility, to optimize usage of the underlying server

hardware resources. This may be driven by scenarios such as:

o Migration from, and shut down of underutilized hosts and consolidating on another more fully utilized

host to reduce energy costs

o Decommissioning one or more older hosts and bringing up workloads on newly added infrastructure

These use cases require “stretching” the tenant’s L2 subnet to connect the servers or VMs since VM-to-VM communications

and/or VM migration between hosts currently requires the hosts to be in the same L2 subnet. Both VNF technologies use

packet encapsulation strategies discussed above, not only to create a larger number of VLANs for subnets, but to also extend

them across L3 networks. Both standards enable multi-tenant networks that can be shared across cloud and on-premise

environments.

It should also be noted that a VM can be moved to a different L2 subnet but doing so today requires reconfiguring its MAC

and IP address. VNF creates a virtual network infrastructure where a VM can be created and moved anywhere without

these limitations.

9 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

The figure below is an example that shows a simplified network topology to better understand the role played by the

Emulex OCe14000 in a VXLAN implementation. Readers are encouraged to read the draft submissions to the Internet

Engineering Task Force (IETF) to understand the building blocks of a VXLAN4 or NVGRE5 enabled network.

Figure 6 – VXLAN Network Topology – The OCe14000 performs VNI encapsulation and de-encapsulation

Stretching an L2 VLAN over Layer 3 with VXLAN or NVGRE is conceptually illustrated below.

Figure 7 –VXLAN or NVGRE networks stretch VLANs to span Layer 3

4 IETF VXLAN draft 5 IETF NVGRE draft

10 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Virtual Network Fabrics – The Path to Software Defined Networking In the age of pervasive virtualization, an IT manager needs the ability to dynamically scale applications to match changing

business needs, streamline the deployment of networking infrastructure and simplify its provisioning, configuration and

management. This application-centric approach to networking requires the entire networking stack to operate with

increased agility and lower operational hurdles and expenses. SDN is an emerging approach to delivering this agility while

ensuring optimal networking performance.

VNF technology is one key enabler for SDN. Today, network connections are managed manually by IT administrators.

Untethering VMs from the underlying physical networks and managing rapid and large numbers of VM relocations requires

a degree of software automation so that VM network connectivity and mobility can be set up and dynamically managed.

Both VXLAN and NVGRE enable this aspect of SDN, and with an intelligent I/O system like the OCe14000 deployed at the

edge, enterprises can leverage the distributed edge processing for more effective I/O and network virtualization. The

OCe14000 platform provides the required encapsulation services required for the VNF’s ID tagging. Emulex VNeX

technology is platform agnostic and provides this service for both VMware vSphere and Windows Server.

Figure 8 – Intelligent I/O Systems Offload Virtual Switching Functions

Supplementing the actual VNF encapsulation technology, an appropriate VNeX I/O programming interface will provide the

needed management and control visibility to the network administrator to rapidly provision and deliver VNF – either

manually or through dynamic interaction between applications and the networking infrastructure.

11 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Emulex Enhances VNF Performance Earlier in this paper, we discussed the role played by various I/O offload technologies that scale networking performance. It

is imperative to maintain these long-standing networking enhancements while onboarding VNFs into the datacenter.

Innovation from Emulex underpins the recent announcement6 of OCe14000 support for the coexistence of hardware

offloads and VXLAN in vSphere 5.5.

This coexistence ensures continued delivery of high performance networking capability while delivering the benefits of VNF

deployments using OCe14000.These benefits include:

Up to 70% better throughput compared to non-offloaded network adapters

Utilization of 50% less CPU resources versus network adapters that use RSS

Optimizing Adapter I/O Bandwidth The performance enhancements in VM compute capabilities and host virtualization density require corresponding

improvements in scaling networking performance. As more workloads are virtualized and consolidated, the network has

the potential to become a performance bottleneck. One of the solutions for this bottleneck is to offload the hypervisor

based software emulation of I/O devices to specialized hardware and device drivers.

Single Root I/O Virtualization The purpose of SR-IOV is to enable a “slice” of the NIC called the virtual function (VF) to be assigned directly to a virtual

machine, thus allowing VM networking I/O to bypass the host and move on a path directly between the VM and the NIC,

eliminating redundant I/O processing in the hypervisor. This, in turn, allows higher I/O performance and lower CPU

utilization as compared to the alternative of software-emulated NIC devices that are implemented in the hypervisor.

SR-IOV is a part of the PCIe specification that enables multiple VMs to directly access the NIC’s physical I/O resources. The

SR-IOV specification includes physical functions (PFs) and VFs and allows a single PCIe compliant network adapter, like the

OCe14000, to appear as multiple physical and virtual devices, using the concept of physical and virtual functions.

Physical function — The PF is a full featured PCIe function of a NIC that supports the SR-IOV interface. The PF

includes the SR-IOV extended capability.

Virtual function — A VF is a lightweight PCIe function on a NIC that supports the SR-IOV interface. VFs are

associated with VMs and are limited to processing I/O streams, and moving data. The number of supported PFs

and VFs can vary; Emulex’s OCe14000 supports up to 8 PFs and up to 63 VFs.

SR-IOV is supported by VMware vSphere 5.1 (and higher), Windows Server 2012(and higher), Red Hat Enterprise Linux 6.0

(and higher) and Solaris.

6 Emulex supports VNF offloads in vSphere 5.5

12 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Figure 8 – Network I/O Path with and without SR-IOV

Benefits of SR-IOV enabled NICs include:

Increased network throughput

Reduced network latency

Reduced host CPU overhead required for processing network traffic

This reduced CPU overhead for network traffic processing frees up resources for increasing VM density, thus optimizing the

usage of host processors.

Universal Multi-Channel and Assigned Bandwidth Universal Multi-Channel (UMC) delivers similar functionality to SR-IOV in that a single PCIe NIC port is split into multiple PCI

functions. The current implementation enables up to four NIC functions per port of an OCe14000 adapter. Each function is

presented to the operating system or hypervisor as a distinct physical port with a separate MAC address – thus allowing up

to 8 functions per dual-port OCe14000 adapter.

Additionally, Emulex OCe14000 converged network adapters maximize configuration flexibility for storage environments

with full protocol offload for FCoE or iSCSI processing. Network administrators can use UMC to partition a single port to

provide specialized NIC functions for different uses such as virtual machine migration and console management, and also

assign specific bandwidth for high-demand applications. Because each function appears to be a separate physical port,

there is no requirement for changes to operating systems or hypervisors.

13 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

Figure 9 –UMC enables partitioning a single NIC port into up to 4 networking ports

The key benefits of deploying Emulex UMC technology include:

Lower Total Cost of Ownership (TCO)

o Consolidates multiple 1GbE adapters, associated cables and switch ports

o Higher VM workload bandwidth allocation to drive higher VM density on host servers

o FCoE or iSCSI storage options save capital expenditures for a dedicated storage adapter

o Lower per-Gb bandwidth cost compared to deploying multiple 1GbE adapters

Optimized I/O Utilization

o Granular bandwidth provisioning minimizes wasted idle bandwidth and waste of dedicated 1GbE

adapters

o Enables Service Level Agreement (SLA) based provisioning and deployment

Simplified Deployment

o UMC is not dependent on specialized OS support

o Works with any 10GbE switch

Continuous Networking Availability A key requirement for a virtualized or a cloud data center network is resiliency through redundancy. With multiple VM

workloads aggregated on a single adapter port, networking I/O is a strategic infrastructure resource. The OCe14000

ensures networking availability for multiple, aggregated VM workloads dependent on a single 10GbE connection for both

VMware vSphere and Windows Server 2012 platforms.

NIC teaming, also called load balance and failover (LBFO) or NIC bonding, is the combination of two or more NICs acting as

one. If one NIC fails the other adapter can still maintain network connectivity, ensuring continuous availability and

resilience to the failure.

14 Emulex White Paper Virtual Network Exceleration – OCe14000 Ethernet Network Adapters

NIC teaming and failover has been supported by Emulex network adapters since ESX 3.5. Windows Server 2012 integrates

and enables vendor agnostic NIC teaming, meaning users are no longer constrained to work with legacy and proprietary

solutions7.

This also means that the NICs from different vendors can be teamed as long as they have passed the Windows Hardware

Qualification and Logo (WHQL) test. The Emulex OCe14000 is qualified for continuous availability solutions in Windows

Server 2012. Microsoft previously did not support NIC teaming8 in Hyper-V.

Conclusion The evolving data center is experiencing architectural shifts from embryonic to pervasive deployment of virtualization

concurrently with the introduction of multi-tenant private, public and hybrid cloud infrastructures.

Emulex’s OCe14000 platform delivers networking capabilities that provide multiple benefits to support this new data

center architecture, including:

Delivering increased VM density to maximize infrastructure ROI

Enabling a secure and scalable cloud multi-tenant infrastructure

Optimizing I/O bandwidth allocation to match workload and management requirements

Ensuring continuous availability for mission critical computing

The Emulex solution includes critical new capabilities such as full hardware support for VNF and other stateless offloads

that ensure optimum utilization of host computing horsepower for maximizing VM density. The OCe14000 also supports

SR-IOV to maximize VM networking I/O throughput without burdening host computing resources further minimizing

adapter TCO. NIC teaming technology has been standardized in the most widely used operating system platforms – users

are no longer constrained to work with legacy NIC solutions from the 1GbE era.

7 Windows Server 2012 (LBFO) Deployment and Management 8 Microsoft NIC teaming support policy prior to Windows Server 2012