nec virtualized epc paper

NEC* Virtualized EPC Innovation Powered by Multi Core Intel® Architecture Processors

NEC* virtualized EPC Innovation Powered by Multi Core Intel® Architecture

2

Table of Contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1 Moves towards network virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Network Virtualization and Software-defined Networking (SDN) . . . . . . . . . . . . . . . . . . . . . . . 4

2.0 NEC virtualized EPC (Evolved Packet Core) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1 Market background and requirement to EPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 NEC virtualized EPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.3 NEC Carrier Grade HyperVisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4 Intel® Data Plane Development Kit Forwarding Technology for NEC vEPC . . . . . . . . . . . 7

3.0 Intel® Virtualization Technology (Intel® VT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.0 Intel® Data Plane Development Kit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.1 Intel® Data Plane Development Kit Architecture Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.2 Intel® Data Plane Development Kit for Virtualized Environment . . . . . . . . . . . . . . . . . . . . . . 10

4.2.1 Intel® Data Plane Development Kit Performance in Virtual Machines . . . . . . . . . . 11

4.2.2 Further Development (OpenVSwitch with Intel® Data Plane Development Kit) . . . . 11


3

AbstractThis paper introduces NEC*’s virtualized EPC (Evolved Packet Core) Innovation that runs on an Intel® architecture server platform. NEC, who has been in the mobile core network business with ATCA hardware equipment for around ten years, is now going to provide carrier-grade, mobile network equipment that will work on the common Intel architecture server platform. Virtualized EPC is the first commercial offering of this effort. Over the past few years, Intel has been promoting a 4:1 workload consolidation strategy that enables multiple workloads (e.g., applications, control plane, data plane and signal processing) to simultaneously run on a single platform. The introduction of NEC’s virtualized EPC solution on Intel architecture demonstrates a leap forward in this direction.

EPC has been traditionally deployed in various platforms using dedicated hardware for specific workloads within the core network. Advances in Intel® microarchitecture, software and networking solutions have made the consolidation of these specific workloads onto a common Intel architecture server platform possible.

This paper will describe how NEC deploys virtualized EPC on an Intel architecture server platform to overcome network virtualization challenges using the Intel® Data Plane Development Kit (Intel® DPDK) and achieve the carrier grade service on a common Intel architecture server platform. Among the discussion items are NEC’s virtualized EPC architecture, Intel architecture, Intel® Virtualization Technology (Intel® VT), Intel® Ethernet Controllers and data plane processing software for Intel architecture.

1.0 IntroductionThe key to enabling successful virtualization of each EPC component is to deliver good networking

performance and maintain virtual machines (VMs) isolation. This section introduces the concepts that are continuously being refined to successfully implement a virtual EPC.

1.1 Moves towards network virtualization

As traffic keeps growing – by as much as ten times in five years – network operators are addressing this demand with continual investments in network appliances. However, this additional equipment requires more space, and the growing varieties of hardware are increasing complexity for operators building up capacity, enhancing performance and replacing equipment at end of life, which happens sooner as technology innovation accelerates.

Virtualization technologies can help reduce the complexities caused by diverse hardware by running vendor applications on common commercial, off-the-shelf (COTS) hardware. In addition, the performance and capability of each application can be dynamically adjusted to satisfy changes in demand. Some of these benefits were highlighted in ETSI’s Network Functions Virtualization (NFV) forum activities, where major global network operators gathered to discuss how to realize such solutions. The solutions are expected to significantly reduce CAPEX and OPEX for operators by simultaneously simplifying capacity management and enhancing network capacity.

NEC architected the individual functions for each EPC node (e.g., Mobility Management Entity (MME), Packet Gateway (P-GW) and Serving Gateway (S-GW)) to run in a virtual machine (VM). Each virtualized function is allocated to a VM. Each VM is totally isolated from the other VMs and runs independently of the other VMs’ required performance. However, all the VMs in a logical node mutually interconnect in order to forward packets correctly. The details of the implementation will be discussed in a later section.


4

1.2 Network Virtualization and Software-defined Networking (SDN)

SDN, which plays a complementary role to network virtualization, facilitates the abstraction of network infrastructure. As a result, network operators can control virtualized network resources across network applications, service types, application types, service providers, etc. by orchestrating various networks from the transport to the application network. The bandwidth and capability of each function can be changed dynamically based on requested application or service, or current demand.

SDN will enhance virtualized network capabilities to monitor and manage VM status, and control the scaling (out/down) of each VM according to the performance and capability demands across multiple services, tenants, applications, etc. Conversely,

network virtualization will maximize the benefits of SDN orchestration through its native ability to provide on-demand flexibility and hardware independence.

2.0 NEC virtualized EPC (Evolved Packet Core)2.1 Market background and requirement to EPC

LTE Core network architecture has fundamentally changed from 2G/3G, and it has shifted to a flat architecture that directly manages each base station (i.e., eNodeB). Simplistically, a MME could represent a busy controller of mobility-related signaling traffic, and a S-GW and a P-GW are likely to focus on data plane (DP) processing. But in reality, their functions are more complex since the S-GW and P-GW also play a vital role in the control plane (CP), handling functions such as inter-eNobeB mobility anchoring.

IndividualFunctions

Intel® VirtualizationTechnology (Intel® VT)for IA-32, Intel® 64 and

Intel® Architecture(Intel® VT-x)

Hypervisor

IndividualFunctions

IndividualFunctions

VirtualMachine 1

VirtualMachine 2

VirtualMachine 3

11

2

3

VM Isolation

VM to VMPacket Forwarding

VM to PortPacket Forwarding

2

3

Intel® VirtualizationTechnology(Intel® VT)

for Directed I/O(Intel® VT-d)

Intel® VirtualizationTechnology

(Intel® VT) for Connectivity(Intel® VT-c)

Figure 1. Image of virtualized network functions


5

Since the naissance of the smart device, mobile network carriers have been facing challenges in expanding performance and enhancing capacities to adapt the network to satisfy the needs and demands of users who expect the LTE experience to be similar to fixed lines or WiFi services. For instance, planning, configuring and tuning the network to handle surging and fluctuating traffic is becoming more complicated. Furthermore, M2M services kicking into high gear in the mobile network will create a new nature of traffic. Network node deployments and configurations could go beyond recognition or financial sense, considering the need to keep up with the traffic using EPC systems bound on static capabilities and a non-flexible performance balance between CP and DP.

It is becoming essential to have elastic network performance and capabilities, as well as flexible performance balance between CP and DP in order to

dynamically handle such traffic fluctuations. Obviously, carrier-grade quality, together with the necessary performance and capabilities, is fundamental to the mobile network.

2.2 NEC virtualized EPC

The NEC vEPC has been realized on commercial off-the-shelf (COTS) servers as virtualized networking functions. Most of NEC vEPC software reuses that of existing ATCA-based, non-virtualized EPC products, which have a rich experience and proven quality in commercial networks. Also, in order to maintain carrier-grade qualities on a virtualization platform and maximize virtualization benefits, NEC CGHV (Carrier-Grade HyperVisor) has been introduced to vEPC, as described in section 2.3, and the Intel DPDK technology, as described in section 2.4.

Figure 2 shows some of the NEC vEPC main features, which realize capacity elasticity for easy scale out

Server Pool

Easy Scale-out

Virtualized Mobile Core (EPC)

COTS server #1

NEC CGHV NEC CGHV NEC CGHV

Ope

nFlo

w C

ontr

olO

penF

low

Con

trol

COTS server #2 COTS server #3

NEC CGHVNEC CGHVNEC CGHV

P/S

GW

(CP

)

MM

E(c

ontr

ol)

P/S

GW

(DP

)

P/S

GW

(DP

)

P/S

GW

(DP

)

MM

E(S

1 IN

TF)

P/S

GW

(DP

)

P/S

GW

(DP

)

MM

E(c

ontr

ol)

Resource ControlResource Control

SDN ControllerSDN Controller

Figure 2. Features of NEC’s vEPC


6

Figure 3. Resource balance between CP and DP for each service

from small start to full system, and for CP and DP performance flexibility. Each virtualized logical node consists of multiple virtual machines (VMs), and each VM is allocated independently to CP or DP of the logical node. This feature enables operators to design each logical node capacity, including CP and DP, independently and flexibly. In addition, the SDN Controller manages and deploys VMs according to service and traffic requirements, and allocates a suitable server, considering each VM’s functional features, necessary resources and system resource availability. VMs are dynamically interconnected for configuring a virtualized logical node through the SDN Controller using OpenFlow technology.

NEC added rich virtualization features to help drastically shorten the time needed to scale capacity and make configuration changes. Using COTS servers for every logical function makes it faster

than dedicated hardware, and the SDN Controller’s sophisticated virtualization functions make it easier and more simplified to procure hardware, and setup and configure additional functions. Leveraging these features, resource allocation suitable for each service and its quick launch are realized as shown in Figure 3.

2.3 NEC Carrier Grade HyperVisor

Generally, many hypervisors are regarded as suitable for IT services, but inappropriate for carrier-grade mobile core networks, which must deliver real-time processes and stable services for a very large number of subscribers, even under heavy load conditions. The NEC Carrier Grade Hypervisor resolves issues related to virtualization with its management functions, real-time performance, high availability, fault tolerance and easy analysis, whose configurations are described in Figure 4.

NEC CGHVNEC CGHV NEC CGHVNEC CGHV

Internet

M2M

Streaming

RAN

RAN

RAN

vMME vP/SGW

CP

DP

CP

DP

CP

DP


7

The NEC Carrier Grade HyperVisor integrates advanced technologies, such as intelligent priority control of CPU resources, disk I/O and network I/O, and memory access optimization to minimize virtualization overhead and hardware conflicts caused by accesses from multiple VMs. Fail-safe logic and program improvements are applied to enhance fault tolerance capabilities, high availability and failover. Analysis features, such as VM resource usage tracing and comprehensive log collection, reduce the time needed for failure analysis and virtualization software troubleshooting.

Furthermore, NEC’s Carrier Grade HyperVisor is based on the Kernel-based Virtual Machine (KVM) and supports its open application programming interface (API), enabling third party applications to maximize operators’ virtualization benefits on the single virtualization platform.

2.4 Intel® Data Plane Development Kit Forwarding Technology for NEC vEPC

The biggest challenge of virtualization was to achieve both a data plane with high-performance forwarding and carrier grade quality (i.e., low latency and jitter) in a virtualized environment.

Figure 4. NEC’s vEPC configuration

NEC adopted the Intel DPDK for its vEPC in order to significantly improve the DP forwarding performance in a virtualization environment. Figure 4 shows the Intel DPDK deployed with a network stack on the guest operating system (OS) for vEPC Data-Plane applications. Furthermore, NEC enhanced the NEC Carrier Grade HyperVisor and the Intel DPDK to overcome DP forwarding quality problems caused by virtualization overhead, conflicts among processes and memory accesses by multiple VMs, thus ensuring low latency and jitter of DP forwarding.

Through the improvements NEC made to its Carrier Grade HyperVisor and the Intel DPDK, high performance DP forwarding with carrier grade quality has been achieved on a virtualization platform.

3.0 Intel® Virtualization Technology (Intel® VT)The fundamental success for Cloud Computing is, arguably, the ability of the system to virtualize a hardware environment. Intel Virtualization Technology offers comprehensive hardware platform capabilities to achieve native performance, with best in class reliability, security and scalability. These capabilities extend from x86 Architectures (Intel® Virtualization

Intel® Data Plane DevelopmentKit (Intel® DPDK)

VMP/S-GW (DP)

Application

Network Stack

Guest OS

VM VM VM VM VM VM

P/S-GW(DP)

P/S-GW(CP)

MME(S1 INTF)

MME(CP) HSS PCRF

NEC CGHV

COTS Server


8

Technology (Intel® VT) for IA-32, Intel® 64 and Intel® Architecture (Intel® VT-x)), Intel® chipsets (Intel® Virtualization Technology (Intel® VT) for Directed I/O (Intel® VT-d)) and Intel Network Devices (Intel® Virtualization Technology (Intel® VT) for Connectivity (Intel® VT-c)).

Intel is continously improving VM performance with each new architecture and process technology.

One of the main causes of performance loss is the overhead required for VM entry and VM exit, also referred to as context switching. Addressing this issue, Figure 6 on the next page shows how context switching latency has been dramtically reduced as new Intel® processor and new Intel microarcitectures have been introduced. This has been possible because of Intel’s ‘Tick Tock’ model, which enables new CPUs to be introduced each year with significant context switching improvement.

Figure 5. Intel® Virtualization Technology (Intel® VT)

Processor Chipset Network

Intel® Virtualization Technology(Intel® VT) for IA-32,

Intel® 64 and Intel® Architecture(Intel® VT-x)

Intel® Virtualization Technology (Intel® VT)

Intel® Virtualization Technology(Intel® VT) for Directed I/O

(Intel® VT-d)

Intel® Virtualization Technology(Intel® VT) for Connectivity

(Intel® VT-c)

Native performance ofvirtualized workloads

with security and reliability

Performance, reliabilityand security through

dedication ofsystem resources

Performance, and scalabilitythrough a dynamically

sharable convergedhigh-capacity interconnect

The success of NEC’s virtualized EPC that is fully deployed on Intel architecture is largely attributed to the Intel Virtualization Technology (Intel VT) advancements. More information on Intel VT can be found on Intel’s online resources listed below.

1) Server Virtualization. Fundamentals of CPU resource virtualization and Intel VT-x hardware accelerators

http://www.intel.com/content/www/us/en/virtualization/processors-extend-virtualization-benefits.html

2) Network Device Virtualization. Fundamentals of network virtualization advancements; example VM Device ques and SR-IOV and Intel VT-c capabilities within Intel® Ethernet Server Adapters

http://www.intel.com/content/www/us/en/virtualization/virtualization-for-improved-network-flexibility.html

https://www-ssl.intel.com/content/www/us/en/virtualization/processors-extend-virtualization-benefits.html?



https://www-ssl.intel.com/content/www/us/en/virtualization/virtualization-for-improved-network-flexibility.html?




9

4.0 Intel® Data Plane Development Kit4.1 Intel® Data Plane Development Kit Architecture Overview

The Intel DPDK is a set of optimized software libraries and drivers that run in a Linux* userspace environment and enable high-performance data plane packet processing on Intel architecture. The Intel DPDK consists of the following components:

Memory/Buffer Manager_ Responsible for allocating pools of objects in memory. A pool is created in huge page memory space and uses a ring to store free objects. It also provides an alignment helper to ensure objects are padded to spread them equally on all DRAM channels. It reduces a significant amount of the

Figure 6. Context Switching Latency in Clocks

time the operating system spends allocating and de-allocating buffers. The Intel DPDK pre-allocates fixed size buffers, which are stored in memory pools.

Queue Manager_ Implements safe lockless queues, instead of using spinlocks, which allow different software components to process packets while avoiding unnecessary wait times.

Flow Classification_ Provides an efficient mechanism for incorporating Intel® Streaming SIMD Extensions (Intel® SSE) to produce a hash based on tuple information so that packets may be placed into flows quickly for processing, thus greatly improving throughput.

8000

7000

6000

5000

4000

3000

2000

1000

0

Cont

ext

Swit

chin

g

Context Switching Latency in Clocks

2005 2006 2007 2008 2009 2010 2011 2012


10

Poll Mode Drivers_ The Intel DPDK includes Linux User Space Poll Mode Drivers for 1 GbE and 10 GbE Ethernet controllers, designed to work without asynchronous, interrupt-based signaling mechanisms, which greatly speeds up the packet pipeline.

Environment Abstraction Layer_ Provides an abstraction to platform-specific initialization code which eases application porting effort. It also contains the run-time libraries for launching and managing Intel DPDK software threads.

Figure 7 shows the components of the Intel DPDK and how they interact directly from user-space with the platform hardware through an abstraction layer. Note, the kernel-space part of the hardware abstraction layer is mostly used for initializing access to hardware and subsequent hardware access is done directly without using the kernel-space abstraction layer (i.e., without involving context switching).

4.2 Intel® Data Plane Development Kit for Virtualized Environment

The Intel DPDK has included support for data plane packet processing within the VM. Among the included features are a VM virtual function driver for Intel® 82599EN/82599EB/82599ES 10 Gigabit Ethernet Controller, virtual NIC and virtual E1000e drivers. Please refer to the official Intel DPDK roadmap for further details.

Figure 7. Intel® Data Plane Development Kit (Intel® DPDK) Overview

Application

User Space

Kernel Space

Linux* Kernel

Environment Abstraction Layer

Buffer Management

Queue/Ring Functions

Flow Classification

NIC Poll Mode Driver

Intel® Data PlaneDevelopment Kit

(Intel® DPDK)

Intel® Data PlaneDevelopment Kit

(Intel® DPDK)

Platform Hardware

ApplicationApplication

EnvironmentAbstraction Layer


11

4.2.1 Intel® Data Plane Development Kit Performance in Virtual Machines

The performance shown below in Figure 8 compares Intel DPDK performance running native (no Virtualization), Intel DPDK with direct passthrough (Intel VT-d), Intel DPDK with SR-IOV (Intel VT-c) and Pure Software (without using Intel DPDK).

The results show the Intel DPDK running on Intel VT-d and SR-IOV achieved native performance for packet sizes 256B and above.

4.2.2 Further Development (OpenVSwitch with Intel® Data Plane Development Kit)

Current ongoing development includes an effort to improve the performance of software switches, which is now gaining prominence and importance. The Intel DPDK will be integrated with OpenVSwitch architecture to replace the current OpenVSwitch forwarding engine with Intel DPDK. This effort will boost the performance of software packet switching and software packet forwarding.

8

Packet Forwarding Performance Estimates for 256B Packet Size

Native Intel DPDK KVM Intel DPDKwith SR-IOV

KVM Intel DPDKwith Intel VT-d

Pure Software

Estimated results reference for Intel® Xeon® processor E5 series with 10GB Ethernet Controller

Mill

ion

Pack

ets/

s

Figure 8. Intel® Data Plane Development Kit Performance Comparison for 256B Packet Size


12

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A “Mission Critical Application” is any application in which failure of the Intel® Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL’S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS’ FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined”. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to:http://www.intel.com/design/literature.htm. All information provided related to future Intel® products and plans is preliminary and subject to change at any time, without notice. All dates provided are subject to change without notice. Intel may make changes to specifications and product descriptions at any time, without notice.

Intel, the Intel logo, Xeon, and Xeon Inside are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Other vendors are listed by Intel as a convenience to Intel’s general customer base, but Intel does not make any representations or warranties whatsoever regarding quality, reliability, functionality, or compatibility of these devices. This list and/or these devices may be subject to change without notice.

Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

This document contains information on products in the design phase of development.

INFORMATION WITH RESPECT TO NEC CORPORATION’S PRODUCTS OR TECHNOLOGIES CONTAINED IN THIS DOCUMENT IS PROVIDED “AS IS”, AND EXCEPT AS OTHERWISE EXPRESSLY PROVIDED IN NEC CORPORATION’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, NEC CORPORATION MAKES NO REPRESENTATION AND WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT OF INTELECTUAL PEOPERTY RIGHTS.

The information contained in this document is subject to change without notice.

NEC Corporation assumes no liability for any technical or editorial errors or omissions that may exist in this document.

The logo of “NEC” is a registered trademark of NEC Corporation in Japan and other countries. Other names and brands used in this document may be trademarks or registered trademarks of their respective owners.

Copyright© 2013 Intel Corporation and NEC Corporation. All rights reserved.

*Other names and brands may be claimed as the property of others.

Printed in USA MS/VC/0613 Order No. 329181-001 US

nec virtualized epc paper

Documents

intel microarchitecture

intel ethernet controllers

virtualized epc innovation

virtual epc

nec virtualized epc52

epc component

data plane processing

virtualized environment104