vcpe and network function virtualisation for...

#clmel

vCPE and Network Function Virtualisation for Enterprises

BRKVIR-2605

Matthias Falkner, Distinguished Engineer, Technical Marketing

© 2015 Cisco and/or its affiliates. All rights reserved.BRKVIR-2605 Cisco Public

Abstract

Network Function Virtualisation is gaining increasing traction in the industry based on the promise of reducing both CAPEX and OPEX using COTS hardware. This session introduces the use-cases for NfV for Enterprise network architectures, such as virtualising branch routers, LISP nodes, IWAN deployments, or enabling enterprise hybrid cloud deployments. The sessions also discusses the technology of NfV from both a system architecture as well as a network architecture perspective. Particular focus is given on understanding the impact of running routing functions on top of hypervisors, as well as the placement and chaining of network functions. Performance of virtualised functions is also discussed.

BRKVIR-2605 Cisco Public© 2015 Cisco and/or its affiliates. All rights reserved.

Agenda

• Introduction & Motivation

• Deployment Models and Characteristics

• The Building Blocks for NfV(today)

• NfV Trade-offs and Research Topics

• Conclusion

8

Introduction and Motivation


Network Functions Virtualisation (NFV)

Announced at SDN World Congress, Oct 2012

• AT&T

• BT

• CenturyLink

• China Mobile

• Colt

• Deutsche Telekom

• KDDI

• NTT

• Orange

• Telecom Italia

• Telstra

• Verizon

• Others TBA…

10


…NFV decouples the network functions such as NAT, Firewall, DPI, IPS/IDS, WAAS, SBC, RR etc. from proprietary hardware appliances, so they can run in software. …..It utilises standard IT virtualisation technologies that run on high-volume service, switch and storage hardware to virtualise network functions..…..It involves the implementation of network functions in software that can run on a range of industry standard server hardware, and that can be moved to, or instantiated in, various locations in the network as required, without the need for installation of new equipment.

What is NfV? A Definition

Sources:

https://www.sdncentral.com/which-is-better-sdn-or-nfv/

http://portal.etsi.org/nfv/nfv_white_paper.pdf

Service

Orchestration

NFVSDN X86

compute

11

https://www.sdncentral.com/which-is-better-sdn-or-nfv/


A. Perceived Benefits for NfV - Architecture

Motivation Description

Reduction of the number of network

elements to manage and deploy

• Integration of network functions into a single system reduces the number

of appliances / NE to manage / configure

• Fewer hardware types to deploy / plan for

Service Elasticity • Deployment of VMs much faster than appliances

• Easy scale up / scale down of services

• Flexible service portfolio (mixing VNFs)

Operational efficiencies through

virtualisation

• Can leverage virtualisation advantages from data centre (vMotion,

dynamic resource scheduling, power management etc) also for VNFs

Reduced complexity for High

Availability

• VMs have a smaller failure domain.

• Stateless deployments become more acceptable, so less complexity

through stateful redundancy deployments

• ISSU simplified by deploying a NEW VM and failing over

12


B. Perceived Benefits for NfV - CAPEX


Deployment of standard x86-based

servers

• Servers considered cheaper than routers / appliances

• Servers already deployed in branch / DC / PoP

Deployment of best-of-breed • Separation of network functions allows best-of-breed services

• eliminates vendor lock-in

• Encourages openness and competition among software vendors

• CAPEX reduction through competition

Cost reduction through economies of

scale

• Deployment of huge server farms in DCs can lead to better resource

utilisation

Simplified Performance Upgrades • Capability to increase performance without forklift upgrades

13


C. Perceived Benefits for NfV - OPEX


Reduction of branch visits • Changes / upgrades in the service can be made in software

• No longer need to swap appliances on-site for service upgrades,

appliance failures

Automated network operations • Virtualisation places focus on automation and elasticity, thus reducing

management

Flexible VNF-based operation • Software upgrades can be done independently per VNF

• VNFs can be placed flexibly in branch, PoP or DC

Elimination / reduction of organisational

boundaries

• IT and network operations align

14

Deployment Models and Characteristics


NfV in the Enterprise – Taxonomy by Function

Enterprise NfV

Network Control

Control Plane vRR, vWLC, vMC, vMS/MRs…

Orchestration, Management &

Policy

Transport

On-premiseBasic

IWAN

CloudPrivate cloud

Public cloud

Network Functions /

Services

On-premise

Router-integrated server

Router + external Server

Server-based (vRouter + VNFs)

CloudPrivate cloud

Public cloud

Hybrid

18

Virtualisation of Control Plane Functions


Shared Services

1. NfV Virtualisation Models: Control Plane Functions

• Virtualisation of Control plane functions

– Route Reflectors

– PfR MC

– LISP MS/MR

– WLC

– …

• Can be on-premise or in larger Enterprise WAN PoPs or in the cloud

– Assuming VNFs are reachable by IP

• CSR 1000v offers functional and operational consistency

– Virtualised IOS XE

WAN

Campus

vWLC vRR

vMS/MR vMC

20


Example: vRR with CSR 1000v

• CSR 1000v offers full IOS XE route-reflector functionality

ASR1001 &

ASR1002-X

(8GB)

ASR1001 &

ASR1002-X

(16GB)

CSR1000v

(8GB)

CSR1000v

(16GB)

RP2 (8GB) RP2 (16GB)

ipv4 routes 7M 13M 8.5M 24.8M 8M 24M

vpnv4 routes 6M 12M 8.1M 23.9M 7M 18M

ipv6 routes 6M 11M 7.4M 21.9M 6M 17M

vpnv6 routes 6M 11M 7.3M 21.3M 6M 15M

BGP sessions 4000 4000 4000 4000 8000 8000

VMs

SP Aggregation

Customer

Premise

SP Core

Data Centre

vRR

21

Virtualising Branch Functions


Virtualisation of Branch Functions

• Current Branch infrastructure often contains physical appliances that complicate architecture

• Typical Appliances vary by branch size– Remote office (1-5 users): firewall

– Small (5-50 users): switched infrastructure, small call control, firewall, IPS/IDS

– Medium (50-100 users): redundancy, local campus, call control, firewall, IPS, IDS, WAAS

– Large (100+ users): redundancy, local campus, call control, firewall, IPS, IDS, WAAS

• …In addition to end-points (Phones, Printers, local storage…)

WAN

Campus /

DC

Branch

CUBECUBE

Branch Appliances

• Router: Routing, ACL, NAT, SNMP..

• Switch: port aggregation

• Services realised with appliances

• Full redundancy

• Could be multi-vendor (Best of breed)

Fib/DSL/Cab.

Fib/DSL/Cab.

23


Branch Virtualisation – On premise Options

Branch

Router + virtualised L4-7 services

• Router performs transport functions (Routing, ACL, NAT,

SNMP..)

• Services virtualized on external server

• Optional redundancy

• VNFs Could be multi-vendor (Best of breed)

F/D

Branch

Fully virtualised Branch

• Physical router replaced by x86 compute

• Both transport and network services virtualised


• VNFs could be multi-vendor (Best of breed)

F/D

1

2

3

WAN

WAN

Branch

Router + integrated L4-7 services

• E.g. ISR + UCS-E

• Router performs transport functions

• Services (Firewall, WAAS..) virtualised on UCS-E


F/D

WAN

24


LISP VPN Gateway

• Challenge

– Extending architecture globally

– Simplicity & flexibility

• Virtual Router Benefits

– Faster provisioning

– Hardware availability

– Rapid reaction to global demand

• CSR 1000v Advantages

– Full-service router supporting LISP and MPLS VPN

– Can be coupled with Encryption

– Consitent L3VPN feature set (IOS XE)

– QoS transparency

LISP – MPLS

Gateway

LISP overlay

Real-WorldExample

25

Cloud Virtualisation


Cloud Virtualisation Categories

• Virtualised routers first deployed in cloud environments

– Bind cloud applications into the enterprise network infrastructure

• New use cases

– Virtualisation of L3 transport for small branches (SOHO)

– Hybrid branch virtualisation environments

27


Application Visibility in the Amazon Cloud

• Cloud network enhanced by sophisticated routing functionality

– Secure connectivity to cloud (encryption)

– VPC to VPC connectivity

– Application Visibility

– WAAS

VPCs are part of enterprise network

End-to-end Cisco network (including

AWS Cloud)

Application Visibility

Remote Sites

& Employees

Enterprise

Data Centre

Public

InternetVPC2

VPC1

VPCs are part of enterprise network

End-to-end Cisco network (including

AWS Cloud)

Application Visibility

28

https://commons.wikimedia.org/wiki/File:Office_building_icon.png

https://commons.wikimedia.org/wiki/File:Office_building_icon.png


WAN

DC

Campus

WAN

DC

Campus

Branch Virtualisation: Cloud Options

L2 Private-cloud Branch – 1:1

• Small branches with low throughput and

no WAAS, Encryption, HA requirements

• Switch: transport, Storm control, L2 COS

• Routing & Services: done in PoP or in SP

DC running on UCS (at PoP or in DC)

• Single tenant, but optionally single-or multi-

site

Routing, QoS,

FW, NAT..

Branch

Branch Routing, QoS,

FW, NAT..

F/D

L3 Private-cloud Branch – 1:1

• L3 router remains in branch but performs

minimal functions

• L4-7 services virtualised in the private

cloud

• Branch router tightly coupled with virtual

router in the private cloud for services

Routing, QoS,

FW, NAT..

Branch

Branch FW, NAT..

F/D

4

5

Suitability for applications with stringent bandwidth / delay /

jitter requirements?

29

The Building Blocks for NfV (Today)


Architecture Building Blocks Enterprise NfV

• A transport network

• Physical Hardware

– X86 servers

– NfV-capable routers

• Virtual Network Functions

– Virtual Routers, Firewalls, NATs…

• Hypervisors / Containers

• Orchestration and Management

• Service Chaining (Optional)

Branch 1

Policy

Orchestration & Management

PHY PHYHost OS

VM1 VM2

PnP

LCMHypervisor

VSwitch

DC

…

WAN Branch N

VMx

31


ETSI NfV Reference Architecture

Source: http://www.etsi.org/deliver/etsi_gs/nfv/001_099/002/01.01.01_60/gs_nfv002v010101p.pdf32

Execution reference points Main NFV reference pointsOther reference points

Computing

Hardware

Storage

Hardware

Network

Hardware

Hardware resources

Virtualisation LayerVirtualised

Infrastructure

Manager(s)

VNF

Manager(s)

VNF 2

OrchestratorOSS/BSS

NFVI

VNF 3VNF 1

Virtual

ComputingVirtual Storage Virtual Network

NFV Management and Orchestration

EMS 2 EMS 3EMS 1

Service, VNF and Infrastructure Description

Or-Vi

Or-Vnfm

Vi-Vnfm

Os-Ma

Se-Ma

Ve-Vnfm

Nf-Vi

Vn-Nf

Vl-Ha

Management

Orchestration

Virtual Network Functions

Hypervisor

Compute Hardware

Virtual Network Functions


Management

& Orchestration

Voice &

Video

Security

Network

Infrastructure

Available VNFs from Cisco for Enterprise (Sample)

Deep Packet

Inspection

(vSCE)

Web Security

(vWSA)

E-Mail Security

(vESA)

Identity Services

Engine

(vISE)

DMVPN

(CSR1Kv)

SSL VPN

(CSR1Kv)

Virtual ASA

Firewall

(ASAv)

NAT

(CSR1Kv)

Virtual Zone

Based Firewall

(CSR1Kv)

IPSec and SSL

VPN

(ASAv)

vNGIPS

(SourceFire)

IPSec VPNs (Flex,

Easy, GET)

(CSR1Kv)

Virtual Router

CE / CPE

(CSR1Kv)

Nexus 1000V

Virtual

Route Reflector

(CSR1Kv, XRv)

CML / VIRL

Wireless LAN

Controller

(WLC/MSE)

Network Analysis

Module (NAM)

Wide Area

Application Service

(WAAS)

AppNav and AVC

(CSR1Kv)

DHCP

(CSR1Kv)

IP SLA

(CSR1Kv)

VXLAN (L2,L3),

OTV, VPLS, LISP

(CSR1Kv)

Virtual

PE/ IP Router

(CSR1Kv)

Cisco VDS-IS

Cisco Unified

Coms Manager,

Presence, Unity

Unified Contact

Centre, CC

Express

CUBE

(CSR1Kv)

Roadmap

Video

Conferencing

(MSE8K)

Enterprise Network

Controller (APIC-

EM)

Prime Performance

Manager, Prime

Analytics

Prime Network

Registrar, IP

Express

Prime

Access Registrar

Prime Fulfillment,

Order Fulfillment

Prime Home

Cisco Prime

Infrastructure,

Provisioning

Prime

Collaboration

Prime Network

Service ControllerUCS Director

Prime Service

Catalog

Intelligent

Automation for

Cloud (IAC)


Cisco Virtual Network Functions

• Adaptations from physical systems / solutions

• Feature and operational consistency between physical and virtual systems

– E.g. CSR 1000v and ASR 1000 / ISR 44xx are all based on the SAME IOS XE

• Exposure of APIs (REST)

• Flexible Licensing models (perpetual, Smart Licensing, Cisco ONE)

• Flexible Performance

– ASAv: {100Mbps, 1Gbps, 2Gbps}

– CSR 1000v: {10Mbps, 50Mbps, 100Mbps, 250 Mbps, 500Mbps, 1Gbps, 5 Gbps, 10Gbps}

– vSCE: 5 Gbps

35

BRKSEC-2762

BRKARC-2010


Cisco CSR 1000V – Virtual IOS XE NetworkingCisco IOS Software in Virtual Form-Factor

Virtualised Networking with Rapid Deployment and Flexibility

IOS XE Cloud Edition

• Selected features of IOS XE based on targeted use cases

Infrastructure Agnostic

Not tied to any server or vSwitch, supports ESXi, KVM, Xen, AMI

Throughput Elasticity

• Delivers 10Mbps to 20 Gbps throughput, consumes 1 to 8 vCPU

Multiple Licensing Models

• Term, Perpetual

Programmability

• RESTful APIs (leverages OnePK) for automated managementServer

Hypervisor

Virtual Switch

VPC/ vDC

OS

App

OS

App

CSR 1000V


Architecture (CSR 1000v) - Virtualised IOS XE

Virtualised IOS XE

Generalised to work on any x86 system

Hardware specifics abstracted through a virtualisationlayer

Control Plane and Data Plane mapped to vCPUs

Bootflash: NVRAM: are mapped into memory from hard disk

No dedicated crypto engine – we leverage the Intel AES-NI instruction set to provide hardware crypto assist.

Boot loader functions implemented by GRUB

Packet path within CSR 1000v

1. Ethernet driver (ingress)

2. Rx thread

3. PPE Thread (packet processing)

4. HQF Thread (egress queueing)

5. Ethernet driver (egress)

Control PlaneForwarding Plane

vNICvCPU vMemory vDisk

Physical Hardware

CPU Memory Disk NIC

Hypervisor (VMware / Citrix / KVM)

Chassis Mgr.

Forwarding Mgr.

IOS

Chassis Mgr.

Forwarding Mgr.

FFP Client / Driver

FFP code Linux Container

37


Technology Package IOS-XE Features

IPBase

Basic Networking: BGP, OSPF, EIGRP, RIP, ISIS, IPv6, GRE, VRF-LITE, NTP, QoS

High Availbility: HSRP, VRRP, GLBP

Addressing: 802.1Q VLAN, EVC, NAT, DHCP, DNS

Basic Security: ACL, AAA, RADIUS, TACACS+

Management: IOS-XE CLI, SSH, Flexible NetFlow, SNMP, EEM, NETCONF

SECIPBase Plus…

Multicast: IGMP, PIM

Advanced Security: Zone Based Firewall, IPSec VPN, EZVPN, DMVPN, FlexVPN

AppX

IPBase Plus…

Advanced Networking: L2TPv3, BFD, MPLS, VRF, VXLAN

Application Experience: WCCPv2, AppXNAV, NBAR2, AVC, IP SLA

Hybrid Cloud Connectivity: LISP, OTV, VPLS, EoMPLS

AX ALL FEATURES

CSR 1000v Feature Support and Technology PackagesREFERENCE

38


Cisco ASAv Firewall and Management Features

Cisco® ASA 9 Feature Set

Cisco

ASAv10

ASAv30

Removed clustering and

multiple-context mode

Parity with all other Cisco ASA platform features

10 vNIC interfaces and VLAN tagging

Virtualisation displaces multiple-context and clustering

SDN (Cisco APIC) and traditional (Cisco ASDM and CSM)

management tools

Dynamic routing includes OSPF, EIGRP, and BGP

IPv6 inspection support, NAT66, and NAT46/NAT64

REST API for programmed configuration and monitoring

Cisco TrustSec® PEP with SGT-based ACLs

Zone-based firewall

Equal-Cost Multipath

Failover Active/Standby HA model

39


Protection Across the Attack Continuum with FirePOWERv

• Virtual machine discovery

• Enforce application policy

• Access control to segment security zones

• Visibility into virtual network communications

• Protect VMs even as the migrate across hosts

• Intrusion prevention without hairpinning

• Single pane-of-glass across physical and virtual networks

• Automated response via Integration with platform security controls

BEFOREDiscover

Enforce

Harden

AFTERScope

Contain

Remediate

Attack Continuum

Detect

Block

Defend

DURING

41


FirePOWERv Virtual Defence Centre• Deployed as virtual appliance

• Inline or passive deployment

• Full NGIPS Capabilities

• Add-on capability

• Control

• Advanced Malware Protection

• URL Filtering

• Deployed as virtual appliance

• Manages up to 25 sensors

• physical and virtual

• single pane-of-glass

DC

Virtual IPS Appliances

42

Hypervisors


CSR 1000v and Hypervisor Processing Relationships

• Example: 3 CSR VMs scheduled on a 2-socket 8-core x86

– Different CSR footprints shown

• Type 1 Hypervisor

– No additional Host OS represented

• HV Scheduler algorithm governs how vCPU/IRQ/vNIC/VMKernelprocesses are allocated to pCPUs

• Note the various schedulers

– Running ships-in-the-night

Pro

ce

ss

Qu

eu

e

HV Scheduler

Core0

pCPU1 pCPU2 pCPU3 pCPU4


vCPU12

vCPU03

vNICn2

VM Kernel1

Core1



vSwitch

VM1(4vCPU CSR 1000v)

CS

RIOSFman /

CManPPE HQF Rx

vCPU01 vCPU1

1 vCPU31 IRQ1 vNIC1

1 VM Kernel1

PP

E

vCPU21 vNICn

1

Guest OS Scheduler

Pkt Scheduler

VM2(1vCPU CSR 1000v)

vCPU02 IRQ2 vNIC1

2 VM Kernel2vNICn2

CS

RIOSFman /

CManPPE HQF Rx

Guest OS Scheduler

Pkt Scheduler

VM3 (2vCPU CSR 1000v)

vCPU03 IRQ3 vNIC1

3 VM Kernel3vNICn3

CS

RIOSFman /

CManPPE HQF Rx

Guest OS Scheduler

vCPU13

Pkt Scheduler

45


Virtual Switches / Bridges

• Virtual switches ensure connectivity between physical interfaces and Virtual Machines

• Can have multiple vSwitches per host

• May have L2 restrictions

• May impact performance

46


Cisco Nexus 1000V

Hypervisor

Modular Switch

…

Linecard-N

Supervisor-1 (Active)

Supervisor-2 (StandBy)

Linecard-1

Linecard-2

Ba

ck P

lane

VEM-NVEM-1 VEM-2

VSM: Virtual Supervisor Module

VEM: Virtual Ethernet Module

VSM-1 (active)

VSM-2 (standby)

Virtual Appliance

Network

Management

Server

Admin

NX-OS

Control Plane

NX-OS

Data Plane

Hypervisor Hypervisor

47

© 2015 Cisco and/or its affiliates. All rights reserved.BRKVIR-2605 Cisco PublicOpenStack Controller

Cisco

Nexus

1000V

VEM

Cisco

Nexus

1000V

VEM

Cisco

Nexus

1000V

VEM

VM VM VM VMVM VM VM VMVM VM VM VM

Cisco Nexus 1000V VSM

Virtual Supervisor Module (VSM)

• Virtual or Physical appliance running Cisco NXOS (supports Hi-availability)

• Performs management, monitoring, and configuration

• Tight integration with management platforms

Virtual Ethernet Module (VEM)

• Enables advanced networking capability on the hypervisor

• Provides each virtual machine with dedicated “switch port”

• Collection of VEMs : 1 virtual network Distributed Switch

KVM KVMKVM

Server Server Server

Cisco Nexus 1000V Deployment Scenario

48


VMWare Scheduler Details

• ESXi scheduler responsible for allocating pCPU to

– vCPU threads

– IRQs

– VM kernel Management

– i/o threads

• Default ESXi behaviour: allocate equal shares of pCPU cycles to each vCPU

– Design goal is fairness across VMs

– vCPU threads have a scheduling entitlement

– If thread has is below its execution entitlement, scheduling priority is raised

– Scheduler also accounts for entitlements and allocations

– Maximisation of pCPU utilisation may have negative throughput effects for VMs (e.g. cache thrash)

• ESXi can also co-schedule related threads (‘relaxed co-scheduling)

– Avoids synchronisation latency for related processes

49


KVM Scheduler Notes

• Each VM appears as a regular linux process to the host OS

Processes can be given relative priorities to influence the scheduler

Implements real-time scheduling extensions

• Linux schedulers generally time-share between processes

Process with the highest current priority gets scheduled onto CPU

Supports dynamic process priorities

Supports pre-emption

• Uses a ‘Completely Fair scheduler (CFS)’ under KVM+RedHat

Includes control groups to allow minimum resource allocations to processes

• Support for live VM Migration

50


Hypervisors vs. Linux Containers

Hardware

Operating System

Hypervisor

Virtual Machine

Operating

System

Bins / libs

App App

Virtual Machine

Operating

System

Bins / libs

App App

Hardware

Hypervisor

Virtual Machine

Operating

System

Bins / libs

App App

Virtual Machine

Operating

System

Bins / libs

App App

Hardware

Operating System

Container

Bins / libs

App App

Container

Bins / libs

App App

Type 1 Hypervisor Type 2 Hypervisor Linux Containers (LXC)

Containers share the OS kernel of the host and thus are lightweight.

However, each container must have the same OS kernel.Containers are isolated,

but share OS and, where

appropriate, libs / bins.

51

I/O Architecture


x86 machine

Host-OS /

KVM

Qemu /

v-Host

tap

vSwitch (OVS) / Linux bridge

NIC driver

Guest-OS

Virtio-net

Guest-OS

Virtio-net

Qemu /

v-Host

tap

AppAppAppAppAppApp

Virtualising I/O – KVM Architecture Example

Hypervisor virtualises the NIC hardware to the multiple VMs

Hypervisor scheduler responsible for ensuring that I/O processes are served.

There is a single instance of physical NIC hardware, including queues, etc.

many to one relationship between the VM’s vNIC and the single physical NIC

One vHost/VirtIO thread used per configured interface (vNIC)

May become a bottleneck at high data rates

NIC port

54


• KVM with OVS consumes a vHost thread per configured VM interface

• The vHost thread is very CPU intensive, requires dedicated physical core

• On 16-core server, can only get 3 CSR1000v (2vCPU, 2 i/f each)

– Cores for CSR: 6

– Cores for VTF: 2

– Cores for vHost: 6

– Free: 2

• Should be considered when service chaining

Hypervisor Traversal Tax: Example KVM with OVS

x86 machine

NIC

Host-OS / KVM

Guest-OS

Open vSwitch

layer-2 sorter / switch / classifier

PF

PF driver

VTF

Virtio-net

Qemu /

vHOST

tap

Guest-OS

Virtio-net

Qemu /

vHOST

tap

CSR

Guest-OS

Virtio-net

Qemu /

vHOST

tap

CSR

Guest-OS

Virtio-net

Qemu /

vHOST

tap

CSR

Guest-OS

Virtio-net

Qemu /

vHOST

tap

CSR

Hypervisor traversal

tax = 8/16 = 50%

May not be

fully utilised!

55


x86 machine

NIC

Host-OS / KVM

Guest-OS Guest-OS Guest-OS

driver driver

I/O Optimisations: Direct-map PCI (PCI pass-through)

• Physical NICs are directly mapped to a VM

Bypasses the Hypervisor scheduler layer

PCI device (i.e. NIC) no longer shared among VMs

Typically, all ports on the NIC are associated with VM

Unless NIC supports virtualisation

• Caveats:

Limits the scale of the number of VMs per blade to ‘number of physical NICs per system’

Breaks live migration of VMs

AppAppAppAppAppApp

AppAppApp

driver

NIC NIC

56


I/O Optimisations: Single Root IO Virtualisation - SR-IOV with PCIe pass-through• Allows a single PCIe devices to appear to be

multiple separate PCIe devicesNIC supports virtualisation

• Enables network traffic to bypass software switch layers

• Creates physical and virtual functions (PF/VF)PF: full featured PCIe

VF: PCIe without configuration resources

Each PF/VF gets a PCIe requestor ID s.t. IO memory management can be separated between different VFs

Number of VFs dependent on NIC (O(10))

• Ports with the same (e.g. VLAN) encap share the same L2 broadcast domain

• Requires support in BIOS/Hypervisor x86 machine

NIC

Host-OS / KVM

Guest-OS Guest-OS Guest-OS

layer-2 sorter / switch / classifier

VF VF VF PF

VF driver VF driver VF driver

AppAppAppAppAppApp

AppAppApp

SR-IOV

Master

Driver

57

Management & Orchestration


CSR 1000v VM Instantiation & Bring-up - Overview

• CSR 1000v VM Instances can be instantiated using the following methods (with possible hypervisor dependencies)– VMWare ESXi: vSphere

– KVM / Xen: Openstack

– Public cloud: Marketplace

• Image Management– VMWare ESXi: vCloud Director

– KVM / Xen: Openstack Glance

– Public cloud: Marketplace

• An new Configuration OVF Tool (COT) is also provided for Cisco VMs

• License management – Smart licensing / Cisco ONE

60


Cisco Prime Infrastructure and CSR 1000v

• Prime Infrastructure supports cross-platform lifecycle management

– Lifecycle -> Design / Deploy / Operate / Report

– Support for templating, workflows

– State-of-the-art GUI with configurable dashboards / dashlets

• Prime Infrastructure NOT helping on managing the servers / hypervisors

– No support for generic server hardware configuration / monitoring

– No hypervisor monitoring / configuration support

• Same Functionality at device level as ISR 44xx and ASR1000

• Prime Network Services Controller (NSC) also supported for Hierarchical, Multi-Tenant Network Services management

61


Prime Infrastructure 2.2 GUI Preview

Prime Infrastructure

Lifecycle Mega-Menus

Prime Infrastructure Configuration

Element Options


Prime Service Catalog

(PSC)User Self-

Service Portal

NFV

Orchestrator

VM and Service

Lifecycle Manager

SDN sub-

system /

SDN

Controller

SDN Virtual

Forwarder

VM & Storage

Orchestrator

DCI

REST API REST API

MP-BGPRestconf

/YangN

etc

onf/

Yang

Or

CLI

Prime Order Fulfillment

or SP’s OSS/BSS

OpenStack APIs

VNF

VNFx86 Server

ESC API VTM API

OpenStack

Elastic Services

Controller (ESC)

Virtual Topology

Controller (VTC)

Service

Assurance

A Framework

enabled by

multiple products

& architecture

OVS

VTF

Cisco NfV Orchestration SolutionOSS

Network Services Orchestrator (NSO)

(Foundation Based on Tail-f NCS)

REST API /

JCloud (Future)

SP WAN

64


Cloud Service OrchestrationOrchestration

WorkflowCatalogPortal / UI / API

VM/Storage Control

Network Control

Network Service Control

Se

rvic

e

Cre

atio

n

Serv

ice

Monitorin

g

Serv

ice

Config

IP

Contr

ol

DC

Netw

ork

Contr

olle

r

WA

N

Contr

olle

r

…

NfV Example Workflow1. Request received

2. Catalog item

3. Defines workflow

4. Workflow calls Service Creation to set up service VMs

5. Service Creation calls to Openstack to set up VMs

6. Openstack sets up VMs

7. Workflow calls to Service Config function to set up services

8. Service Config configures services

9. Workflow calls DC network controller

10. DC network controller configures overlay network

11. Service monitoring tracks availability and performance of service

12. Service Creation manages service elasticity and high availability

Infrastructure

Physical

Network

Virtual

Network

Compute

Storage

Virtual Services

1112

10

1

2 3

4 7 9

5

6

8

68

Service Chaining

© 2015 Cisco and/or its affiliates. All rights reserved.BRKVIR-2605 Cisco PublicUnderlay

Service Chaining Architecture Components

• Virtualisation facilitates a service chaining paradigm– Allows for significant architectural shifts

• Components1. Service classifier (SC): filter flows that enter a particular chain of service functions

2. Loadbalancer (LB): enable elastic capacity expansion / contraction in case the aggregate traffic volume for a service function exceeds its capacity

3. Service function (Sf): VMs or hardware that execute a one or more service functions in the chain

4. Transport Protocol used to carry packets between the loadbalancer / classifier and the service functions

5. Underlay network to get packets from one service function to the next

Service3Service2Service1

Service4Service1SC/LB1,2

3

4

5


Benefits of Service Chaining

• Policy-based execution of service functions: see example on next slide

• Best of breed service functions– Can stitch together service functions from different vendors who each in turn deliver

best-of breed

• Independent troubleshooting and management– Each service function can be configured / managed / upgraded independently

– Allows for different departments to be responsible for a particular service function

– Elastic capacity expansion / contraction of service functions

• Leverage benefits of virtualisation– Fast and flexible introduction / expansion of services

– VM Moves

– Optimal placement of service functions

– Server hardware upgrades

• Flexible service ordering71


Try rendering a business policy like …

All traffic between the Internet & Web front end servers apply:

De/Encryption with highest throughput / low latency and least $$ cost

Copy all “mobile” only transactions to a Big Data analytics system

Perform the copy at most optimal point ($$ cost & least latency impact)

Send all traffic through a SLB+WAF & and IDS

Additionally, deploy this policy with other caveats like:

Service functions are both virtual and physical and vendor neutral

Compute & service elasticity; compute mobility

Practically impossible today!

Why we must Evolve Service Chaining

InternetElastic

SSL

Elastic

LB +

WAF

Elastic

IDS

Elastic

Web FE

Elastic

Copy

Elastic

Analytics

Mobile

72

NfV Trade-Offs and Research Topics


Main Trade-off and Research Areas

1. Cost of NfV solution as a function of performance

2. Trading-off performance for virtualisationflexibility

– Tuning performance may impact virtualisationelasticity

3. Architectural Considerations

– Capacity planning Service Function Chains?

– Orchestration solution?

– High-Availability requirements?

74

CAPEX / OPEX

PerformanceArchitecture


Cost / Performance Trade-offs

• CAPEX Viability for virtualisation may require a minimum VM-packing density on a server

– How many VMs can be deployed simultaneously to achieve a certain CAPEX goal?

– Particularly applicable for Cloud deployment architectures

• What are cost effective deployment models?

– Mixing of application VMs and VNFs on the same hardware?

– Single-tenant / Multi-tenant?

– Hypervisor type?

– Hyperthreading?

– SLA guarantees and acceptable loss rates?

– High-availability requirements and architectures?

75

Architectural Considerations


WAN

Differences Between Cloud and Branch NfV Use-Cases

• Focus on cloud orchestration and virtualisation features

• Mix of applications and VNFs may be hosted in the cloud

• Horizontal scaling -> smaller VM footprints

• Dynamic capacity & usage- / term-based billing

• Focus on replacing hardware-based appliances

• Typically smaller x86 processing capacity in the branch

• NfV applications (Firewall, NAT, WAAS..) may consume large proportion of available hardware resources

– larger VM footprints

• Cloud orchestration and automation has to be distributed over all branches

– integration with existing OSS desirable for migration

UCS

VDI VDI

DB

ERP

Win WinDPI

UCS

IPSFirewall

WAN

DC

Branch

UCS

VDI VDI

DB

ERP

Win WinDPI

UCS

VDI VDI

DB

ERP

Win WinDPI

77


• Deployment of multi-tenant VMs can significantly improve the business case

– Leverage multi-tenancy feature set in IOS XE on CSR 1000v

• Leverages different footprint sizes of CSR 1000v, for example

– Deploy small footprint for single-branch & large footprint for multi-branch

• BUT:

– comes with a different operational model (Need to consider multi-tenancy for on-boarding a new branch)

– Has different failure-radius implications

Single-Branch vs. Multi-Branch VM Deployments

Branch 1

WAN DC / Cloud

Branch N

Branch 1

WAN DC / Cloud

Branch N

79


CSR 1000v as Multi-tenant vCPE - Example

Profile 1 (multi-tenant)

1vCPU CSR – 400 Mbps

200 VRF’s @ 5Mbps/VRF

QOS, DHCP Server, Static Route, IP SLA,

SNMP

Profile 2 (single-tenant)

1vCPU CSR - 50Mbps

QOS, DHCP Server, OSPF, IP SLA,

IGMPv2, PIM SM, SNMP, ACL2

Number of VM instances / server chassis 20 44

Number of branches / VNF instance 40 1

Total number of branches / server blade 800 44

Total aggregate bandwidth / server chassis 8 Gbps 2.2 Gbps

• Multi-tenant CSR 1000v deployed for 5 Mbps ‘vanilla’ branches requiring 5 Mbps each

• Single-tenant CSR 1000v deployed for high-end branches requiring 50 Mbps each

– Note that the 44 VM scenario (Profile 2) is oversubscribed, however the max bandwidth per VM requirement is only 50Mbps

80


VNF High-Availability Architecture Considerations

• Traditional Networking: make all critical network services highly-available

• Active-Standby or Active-active redundancy models

• Stateful redundancy for NAT, Firewall (i.e. stateful services)

• Adds architectural complexity

– HSRP, NSR, Stateful HA features…

• Does a virtualised environment need HA?

– Depends on PIN

• Branch: YES

• Cloud: MAYBE

– Can rely on reload / re-boot of VMs as this happens much faster

– Function of VM scope (cf. single-branch VNFs)

83


Multiple Single-feature VMs (star)

Service Chaining HA

• Many of the functions considered for service chaining are stateful

• VM HA not sufficient

e.g. VMWare HA relies on replay, so crash of the active VM also crashes the standby

• Require a mechanism to copy VM state and deploy and active-active or active-standby model

CSR will support stateful inter-VM HA for NAT/ZBFW in IOS XE 3.13

Need to collect requirements for feature-independent HA architecture

• Impacts placement of active / standby VMs

Same DC? Different DCs?

• LB needs to be re-programmed upon failure

S1act S2act

LB

S2sbyS1sby

Multiple Single-feature VMs (Circular)

S1act S2act

LB

S2sbyS1sby

Single Multi-feature VMs

(S1+S2)act

LB

(S1+S2)sby

S2sby

S2act

S2sby

S2act

(S1+S2)act

(S1+S2)sby

Animated Slide

84

Performance Aspects for VNF Deployments


Performance Aspects for VNF Deployments

• Throughput / SLAs for VNFs are determined by a multitude of factors

– System architecture, in particular I/O

– Hypervisor type (VMWare ESXi, KVM, Microsoft HyperV, Citrix XEN..)

• Throughput can be increased significantly by hypervisor tuning

• Need to determine

– How many VMs to run on a server blade

– Acceptable frame loss rates

86


• VMWare ESXi and KVM schedulers can perform in the same order of magnitude with tuning

– BUT: need to apply tuning recommendations, especially for KVM

– Most impactful tuning: I/O Optimisations (e.g. VM-Fex, SR-IOV)

• KVM shows bottlenecks when untuned

– descriptor ring restriction in KVM limits performance improvements for larger vCPUVMs

Hypervisor Impacts on Performance

CEF ACL NAT Qos Firewall IPSec(100tunnel-SHA)

8vCPU/4GB 2006 1918 1410 2013 1181 1073

4vCPU/4GB 2228 2014 1296 2190 944 510

2vCPU/2.5GB 1895 1466 1188 1554 936 279

1vCPU/2.5GB 999 622 588 622 548 183

0

500

1000

1500

2000

2500

Throughp

ut(M

bps)

ESXiiMIXThroughput(Mbps)with0.01%lossRateacrossFootprints-Uni-D,IOSXE3.13

UCSC200M2:2xIntelXeon2690SandyBridge2.90Ghz8cores,16threads

CEF ACL NAT QOS Firewall IPSec(100tunnel-SHA)

4vCPU/4GB 898 811 720 745 404 187

2vCPU/4GB 851 778 681 712 391 189

1vCPU/4GB 845 761 680 701 380 179

0

100

200

300

400

500

600

700

800

900

1000

Throughput(M

bps)

KVM+UbuntuiMIXThroughput(Mbps)with0.01%lossRateacrossFootprints,IOSXE3.13

UCSC200M2:2xIntelXeon2690SandyBridge2.90Ghz8cores,16threads


Use a Direct path I/O technology (SR-IOV w/ PCIe pass-through) with CPU tuning below! Otherwise the following configurations are recommended:

KVM Performance Tuning Recommendations

Tuning

Recommendation

Details / Commands Tuning

Disable Hyperthreading Can be done in BIOS CPU

Find I/O NUMA Node cat /sys/bus/pci/devices/0000:06:00.0/numa_node

Enable isolcpus run command “numactl -H” CPU

Pin vCPUs ‘sudo virsh vcpupin test 0 6’ CPU

Set CPU in performance Mode run /etc/init.d/ondemand stop. CPU

Set Procsessor into pass-

through

virsh edit <vm name>

add this line <cpu mode='host-passthrough' />

CPU

Disable IRQ Balance run “service irqbalance stop”. CPU

NUMA-aware VM edit vm config by virsh edit <VM name>.

<vcpu placement='static' cpuset='8-15'>1</vcpu>

CPU

IRQ Pinning find specific nic interrupt number from /proc/interrupts. set affinity to other core than

pinned cpu than for CPU and vHost pinning

CPU

89


KVM Performance Tuning Recommendations (cont.)

Tuning

Recommendation

Details / Commands Tuning

Pin vHost processes ‘sudo taskset -pc 4 <process Number>’,

Where <process Number> is found using ‘ps -ef | grep vhost’

I/O

Change vnet txqueue length to

4000

Default tx queue length is 500

‘sudo ifconfig vnet1 txqueuelen 4000’

I/O

Turn off TSO, GSO, RSO, ‘ethtool -K vnet1 tso off gso off gro off’ I/O

Disable KSM echo 0 > /sys/kernel/mm/ksm/run

NOTE: these settings may impact the number of VMs that can be instantiated on a server / blade

90


Quantitative Impact of various hypervisor tuning steps

Sample Results of Different Performance Improvements

defaultw/Hyperthreading HyperthreadingOff vCPUPinningonly Txqueuelenof4000only

Txqueuelenof4000+vCPUPinning+vhostpinning+txo,rxooff+Hyper

threadingOff

AverageThroughputMbps 100% 145% 174% 509% 952%

0%

100%

200%

300%

400%

500%

600%

700%

800%

900%

1000%

AverageThrough

put(M

bps)

SampleImpactwithdifferentHypervisorTuningsKVM+Ubuntu1.0withOVS,2vCPUCSR1000v,XE3.12Engineeringimage,IMIXtraffic,UCS2202.7GHz,0.01FLR

91


SR-IOV Virtualisation Caveats

• vSphere vMotion

• Storage vMotion

• vShield

• NetFlow

• VXLAN Virtual Wire

• vSphere High Availability

• vSphere Fault Tolerance

• vSphere DRS

• vSphere DPM

• Virtual machine suspend and resu

• Virtual machine snapshots

• MAC-based VLAN for passthrough virtual functions

• Hot addition and removal of virtual devices, memory, and vCPU

• Participation in a cluster environment

• Network statistics for a virtual machine NIC using SR-IOV passthrough

• The following features are not available for virtual machines configured with SR-IOV:

92


• Only works for 1vCPU VMs

• Fault Tolerance is not supported or incompatible in combination with

• Snapshots

• Storage vMotion

• Linked Clones

• VM Backups

• Virtual SAN

• Symmetric multiprocessor VMs

• Physical raw disk mapping

VMWare ESXi Fault Tolerance Caveats

• Paravirtualised guests

• NIC Passthrough

• Hot-plugging devices

• Serial or parallel ports

• IPv6

• …

93


HT Oversubscription

Running Multiple VMs on a Server Blade

• Throughput is typically NOT additive as the number of VMs increases on a server blade

– Major bottleneck is I/O

– Hyperthreading and oversubscription effects

– Cache thrashing

• I/O hypervisor bypass techniques improve multi-VM system throughput

– BUT: may impact virtualisation features (Vmotion etc.)

94

0%

100%

200%

300%

400%

500%

600%

700%

800%

1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

1 vCPU VMs

Ag

gre

ga

te T

hro

ug

hpu

t M

pbs

Near linear

performance increase

as VMs are added

due to VMFex with

Direct Path

Hypervisor CPU

contentionVM

Oversubscription

B200 M2, 12 Cores, 2.67 Ghz

VM/FEX & Direct Path

ESXI 5.1

IP Packets CEF IMIX


0

20

40

60

80

100

120

140

160

180

200

0.00

0.40

0.80

1.20

1.60

2.00

2.40

2.80

3.20

3.60

4.00

4.40

4.80

5.20

5.60

6.00

6.40

6.80

7.20

7.60

8.00

8.40

8.80

9.20

9.60

10.00

10.40

10.80

11.20

11.60

Norm

alizedThrough

put(%

toBaselin

e)

%TrafficLossAccepted

Throughputasafunc onofacceptableTrafficLoss(%)

Esxi

1%reference

Log.(Esxi)

Loss Rate Interpretation - Background

• Performance results vary depending on what acceptable frame loss is defined. Typical definitions for loss rates (FLR) range from

– Absolutely 0 packets lost -> Non-drop Rate

– 5 packets lost

– 0.1% of PPS lost

• Small relaxation of FLR definition can lead to significant higher throughput

• Typically FLR Test data reported for 5 packet loss (to account for warmup) with multiple consecutive 1 minute runs

95


Determination of Desired Frame Loss Rate

• Throughput can be affected by definition of acceptable loss rates

• Tests measure % of dropped traffic for various traffic loads

– Offer traffic load -> observe loss -> reduce offered load until desired loss rate reached

• BUT: Difficult to get consistent data across multiple runs.

• How to interpret the right loss-rate?

• Example:

– Highest rate at which LR of 0.01% appears -> 475 Mbps

– Lowest rate below which LR of 0.01% is ALWAYS observed -> 374 Mbps

– Loss rate ‘violations’ at {445, 435, 414, 384} Mbps

Sample Data

REFERENCE

97

Conclusion


Summary

• Introduction & Motivation

• Deployment Models and Characteristics

• The Building Blocks for NfV (today)

• NfV Trade-offs and Research Topics

99


Key Conclusions

1. Network Function Virtualisation is rapidly maturing and enabling first use-cases TODAY for enterprise network functions

– Virtualisation of control plane functions

– Cloud-based network services

2. NfV enables new architectural approaches leading to potential CAPEX and OPEX savings

– Unclear Benefit from replacement of existing transport infrastructure solutions for the sake of it

– Orchestration and Management put into the spotlight

3. Architectural details both at the system and network level need to be well understood and examined

– E.g. Service Chaining

100


Call to Action

• Visit the World of Solutions for

– Cisco Campus – CSR Demo, APIC-EM

– Walk in Labs

– Technical Solution Clinics

• Meet the Engineer

• Lunch time Table Topics

• DevNet zone related labs and sessions

101


Continue Your Education

• Demos in the Cisco Campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

102


Give us your feedback and receive a

Cisco Live 2015 T-Shirt!

Complete your Overall Event Survey and 5 Session

Evaluations.

• Directly from your mobile device on the Cisco Live

Mobile App

• By visiting the Cisco Live Mobile Site

http://showcase.genie-connect.com/clmelbourne2015

• Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected in the World of Solutions

on Friday 20 March 12:00pm - 2:00pm

Complete Your Online Session Evaluation

Learn online with Cisco Live!

Visit us online after the conference for full

access to session videos and

presentations. www.CiscoLiveAPAC.com

http://showcase.genie-connect.com/clmelbourne2015

http://www.ciscoliveapac.com/

vcpe and network function virtualisation for...

Documents