optimizing nfv performanced2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkspm-2029.pdf · nfv...
TRANSCRIPT
• Introduction to NFV
• Performance Optimizations
• I/O Optimizations• Network connectivity
• Virtual Switches
• Pass Through
• Compute Optimizations• NUMA
• Huge Pages
• CPU Pinning
• Conclusion
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Standards-based Architecture
UCS/Nexus
KVM
OpenStack
OpenStack
Controller
ESC
NSO
Enabled by
Tail-fNew or Existing OSS/BSS
ETSI GS NFV 002 V1.1.1 (2013-10)
NFV Reference
Architectural Framework
EPC Policy
EMEM
IMS
EM
BRKSPM-2029 4
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VNF Deployment Options
Bare Metal
• Dedicated hardware
• No virtualization overhead
• High performance
• Easy to secure
Virtual Machines
• Shared host for multiple VNF
• Performance limited by virtualization
• No shared kernel more secure
Containers
• Shared host for multiple VNF
• Bare metal performance
• Shared kernel with host less secure
VNF App
Host kernel
VNF VM
kernel
VNF VM
kernel
Host kernel
VNF
App/OS
VNF
App/OS
Host kernel
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Pain Points & Bottlenecks
Network I/O
CPUMemory
Network
Appliances
Virtual
Network
Functions
BRKSPM-2029 6
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NFV Performance Evolution
7BRKSPM-2029
Virtual Machines
Virtual Switches
PCI Passthru
SR-IOV
NUMA
Huge Pages
CPU Pinning
High Performance
vSwitch
Containers
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Connectivity Options Abound
OVS
OVS
w/DPD
K
Linux
Bridge
Open
Day
Light
Open
Contrail
DPDK
PCI-PT
SR-IOV
PCI-PT
Virtual
switch
Virtual
distributed
switch
VMXNET
Kernel
VMXNET
DPDK
OpenStack VMware
BRKSPM-2029 8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Connectivity in the Cloud
VM VM VM VM
PF
VF VF
VM VM
kerne
l
ov
s
VM VM
kerne
l
ov
s
Virtual Appliance Web-scale
PCI Passthrough
• Entire PCI device passed through to VM
SR-IOV
• PCI device sliced into multiple virtual
devices to be passed to VMs
PCI Passthrough SR-IOV OVS OVS w/DPDK
OVS
• Virtual switch for scale and migration
OVS w/DPDK
• Virtual switch with high speed direct
access to physical NIC
crypto
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
OpenStack Networking – Tenant Networks
OpenStack
Controller
Compute Node Compute Node
L2/L3
Switch
VM VM
Tenant
Networks
• Contained within the cloud
• Access to external networks through the Network Node
• DVR allows L3 agent to reside on each compute node
Network Node
L3DHCP
BRKSPM-2029 12
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
OpenStack Networking – Provider Networks
OpenStack
Controller
Compute Node Compute Node
VM VM
Provider
Networks
L2/L3 Switch
Network Node
DHCP
BRKSPM-2029 13
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Virtio-net
Compute Host
User space
Virtual
Machine
• OVS is the default vSwitch in OpenStack
• Neutron makes it simple to use
• OpenStack OVS adds security
• Security Groups
• Lower performance because of large number of VM Exits
• VM Entry: kernel mode to guest mode transition
• VM Exit: guest mode to kernel mode transition
• Since OVS and Linux Bridge run in kernel space, all I/O from the VM (guest space) requires VM exit / entry
eth1
OVS / LB
Kernel
sockets
virtio_net
vNIC
VM 1
QEMU FE
virtqueue
Kernel & Network
Drivers
Tap Device
QEMU BE
Compute Host
Kernel space
BRKSPM-2029 15
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
vhost-net
• Offers network traffic between the guest and kernel of the host directly
• QEMU is only used for control plane, negotiation, live migration, etc.
igb
eth1
Tap
Device
QEMU BE
QEMU FE
vhost_net
vNIC
virtqueue
Application
Kernel
sockets
Kernel with
OVS Module
Compute
Host
User
Space
Virtual
Machine
Compute
Host
Kernel
Space
BRKSPM-2029 16
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is DPDK?
• Data Plane Development Kit
• Set of data plane libraries and NIC drivers
• Enables a fastpath for high speed packet processing
• Bypasses Linux kernel and network stack
• Can be used on bare metal host or a VM
• Limited security options
Kernel with
Network Drivers
Application Application
DPDK Libraries
Kernel with
Network
Drivers
No DPDK DPDK
user space user space
BRKSPM-2029 17
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Open vSwitch with DPDK
• Includes DPDK libraries in OVS
• OVS uses DPDK fastpath network connection
• Allows direct access from the guest to the host’s user space by-passing QEMU
• QEMU is only used for control plane, negotiation, live migration, etc.
• OVS moves into user space – minimizing context switches
• Easier to work with than SR-IOV or PCI-Passthrough
igb
eth1
OVS
with
DPDK
Tap
Device
Application
QEMU BE
QEMU FE
vhost-user
virtqueue
Kernel
sockets
Kernel &
Network
Drivers
Compute
Host
User
Space
Virtual
Machine
Compute
Host
Kernel
Space
BRKSPM-2029 18
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PCI-Passthrough Networking
• Pass a PCI device directly to VM
• Limited by number of PCI devices
• VM interface gets full resources of device
• Cumbersome to use
• OpenStack does not see the interface
• Cannot enforce security group rules
• Very good for performance interfaces
eth1
Kernel
sockets
VM 1
vNICVirtual
Machine
Compute Host
User space
Compute Host
Kernel space
BRKSPM-2029 20
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Compute Host
Kernel space
SR-IOV
• Singe Root – I/O Virtualization
• Each NIC is a Physical Function (PF)
• Virtualize PFs into multiple Virtual Functions
• VF passed to VM similar to PCI-Passthrough
• VFs share the resources of the PF
• Good for performance interfaces
SR-IOV PF
eth1
Kernel
sockets
Net 2
Virtual Machine 1
Net 1
Kernel
sockets
vNIC
VF Driver
vNIC
VF Driver
VF
Driver
VF
Driver
VF
Driver
Net 1
Kernel
sockets
vNIC
VF Driver
Virtual Machine 2
Compute Host
User space
BRKSPM-2029 21
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
SMP
• Symmetric Multi-Processing
• All CPUs have equal (uniform) access to shared memory bus
• Does not scale well as the system bus gets congested
CPU 0 CPU 1
I/O Shared Memory
CPU 2 CPU 3
CPU 4 CPU 5
system bus
BRKSPM-2029 23
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NUMA
• Non-Uniform Memory Access
• Each CPU in the node has fast access to local memory
• Slower access to remote node’s memory is used less often
• Most of the data used by any CPU will be stored and accessed locally
Local
Memory
system interconnect
Core 1 Core 2
Core 3 Core 4
CPU 0
Core 1 Core 2
Core 3 Core 4
CPU 1
Local
Memory
NUMA Node 0 NUMA Node 1
BRKSPM-2029 24
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Huge Pages
• 4K is default page size, but can support 2MB or 1GB
• Larger pages reduce the number of pages, shrinking page table size
• Smaller table size means faster searches
• Larger pages increase cache hits and reduce cache misses
• Can be locked to prevent swapping
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
4K
1GB 1GB 1GB
Huge Page Table
4K Pages 1GB Pages
BRKSPM-2029 25
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is a Virtual CPU?
• 2 Sockets (physical CPUs)
• 8 Cores per Socket
• 2 Threads per Core (hyper-threading)
T1
T2
T1
T2
T1
T2
T1
T2
Core 0 Core 1 Core 2 Core 3
T1
T2
T1
T2
T1
T2
T1
T2
Core 4 Core 5 Core 6 Core 7
T1
T2
T1
T2
T1
T2
T1
T2
Core 0 Core 1 Core 2 Core 3
T1
T2
T1
T2
T1
T2
T1
T2
Core 4 Core 5 Core 6 Core 7
Socket 1 Socket 2
2 x 8 x 2 = 32 vCPU
Server with dual Intel E5-
2690 8-core processors
w/Hyper-threading
BRKSPM-2029 26
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
CPU Pinning
T1
T2
T1
T2
T1
T2
T1
T2
Core 0 Core 1 Core 2 Core 3
T1
T2
T1
T2
T1
T2
T1
T2
Core 4 Core 5 Core 6 Core 7
T1
T2
T1
T2
T1
T2
T1
T2
Core 0 Core 1 Core 2 Core 3
T1
T2
T1
T2
T1
T2
T1
T2
Core 4 Core 5 Core 6 Core 7
Socket 1 Socket 2
HypervisorVM
2 vCPUs
VM
4 vCPUs
Compute Host
BRKSPM-2029 27
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Combining Optimizations
• Configure NUMA for fast local memory access for CPUs
• Enable huge pages for fast memory searches
• Pin the vCPUs to physical CPU cores/threads
NUMAHuge Pages
CPU Pinning
BRKSPM-2029 28
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Summary
Area of
PerformanceScalability
Migration
SupportPerformance Security
Virtio Network Good Yes Low High
PCI
PassthroughNetwork Poor No Very High Low
SR-IOV Network Good No High Low
High Speed
vSwitchNetwork Good Yes High High
Description
NUMA Each NUMA node consists of one or more CPU with high speed access to local memory
Huge Pages Allows for a smaller table of larger memory pages for fast lookup
CPU Pinning CPU resource management to keep applications on dedicated cores
BRKSPM-2029 29
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Glossary• DPDK – Data Plane Development Kit
• ESC – Elastic Services Controller
• ETSI – European Telecommunications Standards Institute
• KVM – Kernel-based Virtual Machine
• NFV – Network Function Virtualization
• NFVI – NFV Infrastructure
• NSO – Network Services Orchestrator
• NUMA - Non-uniform Memory Access
• OVS – Open Virtual Switch
• PCI - Peripheral Component Interconnect
• SMP - Symmetric Multi-Processing
• SR-IOV – Single Root I/O Virtualization
• VNF – Virtual Network Function
• VNFM – VNF Manager
BRKSPM-2029 30
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
BRKSPM-2029 31
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
US Ciscolive 2016 @ Las Vegas- SP Mobility SessionsSession ID Time Session Title Speakers
Sunday (10 July 2016)
LTRSPM-2022 01:00 – 05:00 PM Virtual Packet Core Orchestration: Spring to Life Gateways and Services Julie Ann Connary, Amir Ahmadi
Monday (11 July 2016)
BRKSPM-2125 08:00 - 10:00 AM Virtualizing Cisco Mobile Packet Core Aeneas Dodd-Noble, Vivek Agarwal
BRKSPM-2071 01:00 - 03:00 PM 5G Technology Updates Paul Polakos, Prakash Suthar
BRKSPM-2029 04:00 – 05:30 PM Optimizing NFV Performance Ali Bokhari
BRKSPM-2022 04:00 – 05:30 PM Using Telco NFV to Deploy Mobility Networks Dave Clough
BRKSPM-2130 04:00 – 05:30 PM Mobile Packet Core (ASR5500) Troubleshooting Guilherme Correia, Rama Ramachandran
Tuesday (12 July 2016)
BRKSPM-2128 08:00 - 10:00 AM Radio Access Network Virtualization Mark Grayson, Oliver Bull
BRKSPM-2026 01:30 - 03:00 PM AT&T Case study for Managing Customer Experience in Mobile Networks Jeronimo Diez De Sollano, Anwin Kallumpurath
BRKSPM-2065 04:00 – 05:30 PM Mobile and Enterprise Security for the Age of Ubiquitous Encryption Humberto La Roche, Christopher ORourke
BRKSPM-2127 04:00 – 05:30 PM Design & Deploying Trusted and Un-Trusted VoWiFi Venkata Reddy Kasu, Arun Gunasekaran
Wednesday (13 July 2016)
BRKSPM-2129 08:00 - 10:00 AM Deploying 4G/LTE branch routers for IOT solutions Vivek Datar, David Mindel
PLNSPM-2021 01:30 – 03:00 PM Network Infrastructure and Operation Transformation Anand Malani, Scott Clark, Vilma Stoss, Rich Plane, Mark Ghattas (Moderator)
BRKSPM-2028 04:00 – 05:30 PM Design and Deployment of Overlay Services on LTE Network Tom Redman
BRKSPM-2122 04:00 – 05:30 PM Virtualizing Mobile Service Providers Transport Network Ravi Narahari, Abbas Abidi
Thursday (14 July 2016)
BRKSPM-2121 08:00 - 10:00 AM Cisco SON and Advanced Analytics Oliver Bull, Ashish Bansal
BRKSPM-2126 04:00 – 05:30 PM Analytics for Large Connected Venues: App Integration with Wi-Fi Infrastructure Matt Swartz, Joshua Suhr
PSOSPM-2505 01:00 – 02:00 PM Cisco Hosted Mobility (Mobility as a Service) Mike Shomaker
WISP Lab (11-14 July)
LABSPM-2011 10:00 - 6:00 PM Cisco Virtualized Packet Core Installation on VMware Amir Ahmadi, William Pedraza
LABSPM-2012 10:00 - 6:00 PM Cisco Virtualized Packet Core Installation on Openstack Christopher Ove, Guilherme Correia
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33Presentation IDPresentation ID
Please join us for the Service Provider InnovationTalk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKSPM-2029 34