summit 16: the hitchhiker/hacker's guide to nfv benchmarking
TRANSCRIPT
June 20–23, 2016 | Berlin, Germany
The Hitchhiker / Hacker’s guide to NFV Benchmarking
Al Morton, Trevor Cooper and Maryam Tahhan
Introduction
The landscape of emerging NFV benchmarking can be very
confusing to navigate.
Goal is to explain the history and state-of-art NFV benchmarking in
ETSI NFV, IETF and OPNFV
Projects
YardstickVerification of the infrastructure compliance when running VNF applications.Umbrella project for performance testing
Storperf Storage performance testing
VSperf Characterize the performance of virtual switches for Telco NFV use cases.
CPerf Controller performance testing
CSIT (Continuous System Integration and Testing)
Development of software code for fully automated VPP code testing, functionality, performance, regression and new functions.
Standards / specifications
Projects Methodologies Tools Workloads
OPNFV Release B Test Tools
Yardstick
Storeperf CPERF VSPERF QTIP
Standards / specifications
Projects Methodologies Tools Workloads
OPNFV Release B Test Coverage
Standards / specifications
Projects Methodologies Tools Workloads
OPNFV Release B Test Coverage (ETSI-NFV Domains)
Yardstick, VSPERF, CPERF, STORPERF, Functest and QTIP
Standards / specifications
Projects Methodologies Tools Workloads
Other Tools: Traffic Generators
Traffic Generator Open Source/Proprietary Hardware/Software
IXIA Proprietary SW/HW
Spirent Proprietary SW/HW
Xena Proprietary SW/HW
DPDK pktgen Open Source SW
Moongen Open Source SW
TREX Open Source SW
pktgen Open Source SW
LAN Tornado RFC 2544 Proprietary SW
Seagull Open Source SW
Standards / specifications
Projects Methodologies Tools Workloads
Other Tools/Testing efforts
Testing Efforts
• CSIT
• “NFV Performance Benchmarking for vCPE” test
• EEMBC creates their own tools for the benchmark specs they develop (Commercial)
• Canonical VNF Performance Interoperability Lab (V-PIL)
SDN Tools:
• Cbench (OS) CBench is a somewhat classic SDN controller benchmark tool. It blasts a controller with OpenFlow packet-in messages and counts the rate of flow mod messages returned.
• WCBench: WCBench consumes CBench as a library, then builds a robust test automation, stats collection and stats analysis/graphing system around it.
• ClassBench-ng: Test utility for generation of firewall/OpenFlow rules based on original (no longer maintained) Classbench.
• NSTAT (OS): Goes beyond with Multi-Threaded Cbench, adds monitoring OF traffic, switch and controller scale testing.
• Veryx Technologies: Commercial tool, measures metrics in IETF Draft
• EXFO: Commercial tools, Load testing of IMS and EPC virtual network functions (VNF
• Rally, Netperf (TCP_CRR Test), iperf.
• ODL : OpenFlow protocol performance measurements in OpenDaylight.
Standards / specifications
Projects Methodologies Tools Workloads
Workloads
VNF Use Case Brief Description Pros Cons Details
Clear Water IMS
IMS Control Plane Clear Water is an open source implementation of IMS (the IP Multimedia Subsystem) designed to provide voice, video and messaging services. Clear water handles only control plane traffic for sessions.
Open source, All in one and Scalable VNF
Available in AMI and OVI formats, Supports Openstack, All In one Install is limited to fixed domains and is recommended not to use this, Multi-instance VNF Image requires Chef integration, however, scripts are provided by Clear Water for IMS deployment
IMF Use Case with Clear Water
vFW Virtual Firewall (canonical VNF)
Open Source (GNU GPL)
IPTables
vAAA Virtual AAA (canonical VNF)
Open Source (BSD) Free Radius
Emerginov IMS control planned - like integration of open source solutions (HTTP/SIP proxies, Media serveur,..) managed by Puppet
Open Source Emerginov
vRouter/vCPE OpenWRT OpenWRT is frequently used as virtual CPE
Open Source OpenWRT images for OpenStack
vIDS Snort Intrusion Detection System Open Source www.snort.org
Traffic Control vCDN Content Delivery Network Open Source Traffic Control on github
OpenSIPS vSBC, SIP server Open SIP Server, which is adaptible and can be used as a vSBC
OpenSIPS website
Standards / specifications
Projects Methodologies Tools Workloads
What is Benchmarking?
Benchmark Definition in IETF
A standardized test that serves as a basis for performance evaluation and comparison.
13
STANDARDIZED
• All aspects of the testing conditions are sufficiently specified and controlled such that the results:
• Are informative.
• Can be reported unambiguously.
• Are repeatable.
BASIS FOR PERFORMANCE
EVALUATION
• The tests measure some performance characteristic that:
• Is relevant and meaningful to users.
• Provides insight to improve an inter-networking technology.
BASIS FOR COMPARISON
• The tests can be:
• Performed on different vendors' devices without modification.
• Observed using signals that are external to the device/system.
Benchmarks are NOT
• Functional tests.
• PASS/FAIL criteria.
• Performed on live networks, or performed with live network traffic.
http://memberfiles.freewebs.com/94/67/101646794/photos/undefined/E_Marvin%20The%20Paranoid%20Android(THGTTG_Garth%20Jennings_2005)-3.jpg
NFV Benchmarking Landscape
Standards / specifications
• IETF
• ETSI NFV
• Other SDOs/Open Source projects
OPNFV Projects
• Yardstick
• VSPERF
• CPERF
• STORPERF
Methodologies
• Traditional PNF benchmarking
• Evolving methods to Benchmark Virtual Network Functions
• Benchmarks of new Infrastructure Components
Tools
• Test Frameworks
• Traffic generators
Workloads
• Stimulus
• Compute/Memory
• VNFs
Let’s break it down….
Standards / Specifications
SDO
sBMWG
Internet drafts
RFCs
Service Quality Metrics
TST WG WIs
Other SDOs/Open Source projects
IEEE
EEMBC (Cloud DC, Streaming/Web, Bug Data)
Standards / specifications
Projects Methodologies Tools Workloads
Other SDOs/Open
Source projects
BMWG, since 1989RFC 1242 (was draft-ietf-bmwg-terms) Benchmarking Terminology for Network Interconnection Devices
RFC 1944 (was draft-ietf-bmwg-methodology) Benchmarking Methodology for Network Interconnect Devices
RFC 2285 (was draft-ietf-bmwg-lanswitch) Benchmarking Terminology for LAN Switching DevicesRFC 2432 (was draft-ietf-bmwg-mcast) Terminology for IP Multicast BenchmarkingRFC 2647 (was draft-ietf-bmwg-secperf) Benchmarking Terminology for Firewall PerformanceRFC 2761 (was draft-ietf-bmwg-atm-term) Terminology for ATM BenchmarkingRFC 2889 (was draft-ietf-bmwg-mswitch) Benchmarking Methodology for LAN Switching DevicesRFC 3116 (was draft-ietf-bmwg-atm-method) Methodology for ATM BenchmarkingRFC 3133 (was draft-ietf-bmwg-fr-term) Terminology for Frame Relay BenchmarkingRFC 3134 (was draft-ietf-bmwg-atm-term-abr) Terminology for ATM ABR BenchmarkingRFC 3222 (was draft-ietf-bmwg-fib-term) Terminology for Forwarding Information Base (FIB) based Router PerformanceRFC 3511 (was draft-ietf-bmwg-firewall) Benchmarking Methodology for Firewall PerformanceRFC 3918 (was draft-ietf-bmwg-mcastm) Methodology for IP Multicast BenchmarkingRFC 4061 (was draft-ietf-bmwg-ospfconv-intraarea) Benchmarking Basic OSPF Single Router Control PlaneRFC 4062 (was draft-ietf-bmwg-ospfconv-term) OSPF Benchmarking Terminology and ConceptsRFC 4063 (was draft-ietf-bmwg-ospfconv-applicability) Considerations When Using Basic OSPF Convergence BenchmarksRFC 4098 (was draft-ietf-bmwg-conterm) Terminology for Benchmarking BGP Device Convergence in the Control PlaneRFC 4689 (was draft-ietf-bmwg-dsmterm) Terminology for Benchmarking Network-layer Traffic Control MechanismsRFC 4814 (was draft-ietf-bmwg-hash-stuffing) Hash and Stuffing: Overlooked Factors in Network Device BenchmarkingRFC 4883 (was draft-ietf-bmwg-benchres-term) Benchmarking Terminology for Resource Reservation Capable RoutersRFC 5180 (was draft-ietf-bmwg-ipv6-meth) IPv6 Benchmarking Methodology for Network Interconnect Devices ErrataRFC 5695 (was draft-ietf-bmwg-mpls-forwarding-meth) MPLS Forwarding Benchmarking Methodology for IP Flows ErrataRFC 6201 (was draft-ietf-bmwg-reset) Device Reset CharacterizationRFC 6412 (was draft-ietf-bmwg-igp-dataplane-conv-term) Terminology for Benchmarking Link-State IGP Data-Plane Route ConvergenceRFC 6413 (was draft-ietf-bmwg-igp-dataplane-conv-meth) Benchmarking Methodology for Link-State IGP Data-Plane Route ConvergenceRFC 6645 (was draft-ietf-bmwg-ipflow-meth) IP Flow Information Accounting and Export Benchmarking MethodologyRFC 6815 (was draft-ietf-bmwg-2544-as) Applicability Statement for RFC 2544: Use on Production Networks Considered HarmfulRFC 6894 (was draft-ietf-bmwg-protection-meth) Methodology for Benchmarking MPLS Traffic Engineered (MPLS-TE) Fast Reroute ProtectionRFC 6985 (was draft-ietf-bmwg-imix-genome) IMIX Genome: Specification of Variable Packet Sizes for Additional TestingRFC 7501 (was draft-ietf-bmwg-sip-bench-term) Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and RegistrationRFC 7502 (was draft-ietf-bmwg-sip-bench-meth) Methodology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and RegistrationRFC 7640 (was draft-ietf-bmwg-traffic-management) Traffic Management BenchmarkingRFC 7654 (was draft-ietf-bmwg-issu-meth) Benchmarking Methodology for In-Service Software Upgrade (ISSU)RFC 7747 (was draft-ietf-bmwg-bgp-basic-convergence) Basic BGP Convergence Benchmarking Methodology for Data-Plane Convergence
Standards / specifications
Projects Methodologies Tools Workloads
BMWG Products in Summary
• RFC 1242 and RFC 2544, Throughput and Latency
• LAN Switches, Firewalls, Multicast
• ISIS, OSPF, and BGP Convergence
• MPLS, MPLS-TE FRR, Flow Export, IPv6
• RFC 2544 Applicability Statement
• IMIX Genome
• SIP Device Benchmarking
• Traffic Management features
• In-Service Software Upgrade
• Basic BGP Convergence
BMWG NFV Internet Drafts
Internet Draft Link
Data Centre Benchmarking Methodology draft-ietf-bmwg-dcbench-methodology-02
Data Centre Benchmarking Terminology draft-ietf-bmwg-dcbench-terminology-05
Benchmarking Methodology for SDN Controller Performance
draft-ietf-bmwg-sdn-controller-benchmark-meth-01
Terminology for Benchmarking SDN Controller Performance
draft-ietf-bmwg-sdn-controller-benchmark-term-01
Considerations for Benchmarking Virtual Network Functions and Their Infrastructure
draft-ietf-bmwg-virtual-net-02
Considerations for Benchmarking High Availability of NFV Infrastructure
draft-kim-bmwg-ha-nfvi-01
Benchmarking Methodology for EVPN draft-kishjac-bmwg-evpntest-00
VNF Benchmarking Methodology draft-rosa-bmwg-vnfbench-00
Benchmarking Virtual Switches in OPNFV draft-vsperf-bmwg-vswitch-opnfv-02Standards /
specificationsProjects Methodologies Tools Workloads
ETSI NFV Testing Specifications I
• Specifies performance benchmarking metrics for virtual switching
• References: IETF RFC 2544, : IETF RFC 2679, : IETF RFC 2680, : IETF RFC 3511, : IETF RFC 4737, : IETF RFC5481, IETF RFC 6703, IETF RFC 6049, IETF RFC 7348, IETF RFC 6815, IETF RFC 6985, draft-ietf-bmwg-ipsec-term-12.txt, ipsec-meth-12.txt, draft-ietf-bmwg-virtual-net-01.txt, draft-vsperf-bmwg-vswitch-opnfv-01.txt.
ETSI GS NFV-IFA 003: vSwitch Benchmarking and Acceleration Specification
• Provides guidelines for test plans that assess different approaches to defining SDN Applications, different ways of arranging and federating SDN Controllers, and arrangements of network switching/forwarding functions (both physical and virtual) to create the various path-implementations between and among NS Endpoints and VNFs
ETSI GS NFV-TST004: Guidelines for Test Plan on Path Implementation through NFVI (in-progress)
Standards / specifications
Projects Methodologies Tools Workloads
ETSI NFV Testing Specifications II
• Provides a list of features which the performance and portability templates (Virtual Machine Descriptor and Compute Host Descriptor) should contain for the appropriate deployment of Virtual Machines over a Compute Host (i.e. a "telco datacentre").
• In addition, the document provides a set of recommendations and best practises on the minimum requirements that the HW and hypervisor should have for a "telco datacentre" suitable for data-plane workloads
• References: IEEE 1588, IETF RFC 2544, IETF RFC 2889, IETF RFC 3393, IETF RFC 3511, IETF RFC 3918.
ETSI - GS NFV-PER 001: NFV Performance & Portability Best Practise
• Enumerates metrics for NFV infrastructure, management and orchestration service qualities that can impact the end user service qualities delivered by VNF instances hosted on NFV infrastructure.
• References: IETF RFC 2330, IETF RFC 6390, ISO/IEC 15939, ITU-T I.356, ITU-T M.3341, ITU-T.1543, NIST SP 800-145.
ETSI - GS NFV-INF 010: NFVI Service Quality Metrics
Standards / specifications
Projects Methodologies Tools Workloads
ETSI NFV Testing Specifications III
• Recommendations for lab validation of VNFs, their interaction with the NFV functionalblocks and the NFV blocks themselves.
ETSI GS NFV-TST001: PRE-DEPLOYMENT TESTING
• Study how interoperability test methodology can be applied to NFV by analysing thefunctional blocks and interfaces defined within the NFV architecture
ETSI GS NFV-TST002: INTEROPERABILITY TESTING METHODOLOGY
Standards / specifications
Projects Methodologies Tools Workloads
Methodologies
• Traditional: Physical Network Function (PNF) Benchmarking
• Evolving methods to Benchmark Virtual Network Functions (VNF)
• Benchmarks of new Infrastructure Components• SDN Controllers and Virtual Switches
Standards / specifications
Projects Methodologies Tools Workloads
Benchmarking Physical Network Devices (Functions)
Test Generator
Test ReceiverDUT
CONFIG Isolated Test Environ.
Test Generator
Test Receiver
Isolated Test Environ.
Easy Calibration and Resource tracking
• How Fast? • in Packets or Bytes?
• How much Latency?
• How much Capacity?
• How many Sessions?
• How Quick is Recovery?
and importantly
• How do Vendor A & B Compare?
Standards / specifications
Projects Methodologies Tools Workloads
But can we still apply this methodology in the context of NFV benchmarking?
Standards / specifications
Projects Methodologies Tools Workloads
Evolving methods to Benchmark Virtual Network Functions (VNFs)
• System Under Test is different
• Workload = Stimulus and SUT
Standards / specifications
Projects Methodologies Tools Workloads
Benchmarking Virtual Network Functions (VNF + NFVI)
• SUT Configuration is also different
27
Test Generator
Test Receiver
SUT
CONFIG
Isolated Test Environ.
VM - VNFVirtual
Resources
General-Purpose Compute Platform
CONFIG
CONFIG
Operating System/Other SW CONFIG
Standards / specifications
Projects Methodologies Tools Workloads
Test Configuration Parameters
28
General purpose platform:• Number of server blades (shelf occupation) • CPUs/cores • Storage system • Memory• BIOS Configuration
Configurations that support the VNF: • Hypervisor • Virtual Machine (and resources allocated to the VM)• Infrastructure Virtual Network
The VNF itself: • Specific function being implemented in VNF • Number of VNF components in the service function chain • Number of physical interfaces and links transited in the service
function chain
Note: This is a PARTIAL list of items that need to be configured and reported with the test results!
Standards / specifications
Projects Methodologies Tools Workloads
More Benchmarking Considerations
29
• Comparison with Physical Network Functions
– Re-use of existing benchmarks, for a fair comparison.
• Continued Emphasis on Black-Box Benchmarks
– Auxiliary Metrics are tempting
– Supply both, may provide useful insight
• New Benchmarks for a Dynamic World
– Time to deploy VNFs, Time to Migrate, Power Consumption
• Assessment of Benchmark Coverage
Assess Benchmark Coverage:3 x 3 became 3 x 4 Matrix
30
\ Criteria\
Functions\SPEED ACCURACY RELIABILITY SCALE
Activation /Creation / Setup
Operation
De-Activation / Deletion / Take-Down
SDN Controller Benchmarks
Speed Scalability Reliability
Setup 1. Network Topology Discovery.2. Reactive Path Provisioning Time. 3. Proactive Path Provisioning Time.4. Reactive Path Provisioning Rate .5. Proactive Path Provisioning Rate.
1. Network Discovery Size
Operational 1. Asynchronous Message Processing Rate.
2. Asynchronous Message Processing Time.
1. Control Sessions Capacity.2. Forwarding Table
Capacity.
1. Network Topology Change Detection Time.
2. Exception Handling.3. Denial of Service
Handling4. Network Re-Provisioning
Time
Teardown 1. Controller Failover Time.
31
We now also have to benchmark new Infrastructure components- SDN Controllers- Virtual Switches
NFV Benchmarking Gaps
• Standards for characterizing Network Services performance in realistic/scaled deployment scenarios
• Consistent and extensible architecture for NFV performance testing including MANO
• Representative Open Source workloads (Compute/memory and Traffic)
• Lifecycle performance metrics and methodologies
Closing the Gaps
• Develop APIs and models for configuration of workloads (Compute/memory and Traffic)
• Automate for testing in diverse environments and deployment scenarios
• Develop Packet generator features for new protocols and traffic profiles
• Unify performance testing frameworks in OPNFV
Reaching our Destination
Netw
ork S
ervic
e P
erfo
rm
an
ce
Community test tools and infrastructure
Test Dashboards
Database of test results
Summary
There is an ongoing effort to ensure alignment across SDOs and Open Source to help converge on standardized tests that serve as a basis for NFV performance evaluation and comparison.
Questions?
http://fordprefectstowel.weebly.com/uploads/2/5/2/3/25231015/6078343_orig.png
http://littlefun.org/uploads/5269efe1e691b24d64aec592_736.jpg
Back up
References1. http://tolly.com/ts/2011/Blade/G8264/Tolly211108BLADERackSwitchG8264.pdf
2. https://tools.ietf.org/html/draft-ietf-bmwg-virtual-net-02
3. Intel® Open Network Platform Release 2.0 Performance Test Report (see legal information p.76) https://01.org/packet-processing/intel%C2%AE-onp
4. https://wiki.opendaylight.org/view/File:Opendaylight-test.pptx
5. https://tools.ietf.org/html/draft-ietf-bmwg-sdn-controller-benchmark-term-01
6. http://tools.ietf.org/html/draft-vsperf-bmwg-vswitch-opnfv-02.txt
7. http://bsdrp.net/documentation/technical_docs/performance
8. http://data.guug.de/slides/lk2008/10G_preso_lk2008.pdf
9. http://www.telematica.polito.it/oldsite/courmayeur06/papers/06-A.2.1.pdf
10. https://www.ietf.org/proceedings/89/slides/slides-89-bmwg-6.pdf
11. http://www.lightreading.com/ethernet-ip/40-gig-router-test-results/d/d-id/608230
12. https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test#Actively_Used_or_Under_Active_Development
13. http://www.ixiacom.com/sites/default/files/resources/case-study/benchmarking-opensource-sdn.pdf
14. https://groups.google.com/a/onosproject.org/forum/?hl=en#!topic/onos-co/qGg2LAnUxLA
37
Evolving methods to Benchmark Virtual Network Functions (VNFs)
• System Under Test is different
• Provisioning VNF and Virtual Network
• COTS configuration
• Virtualization layer choices
• Shred resources
• Capacity considerations
• Performance metrics
Standards / specifications
Projects Methodologies Tools Workloads
System Under Test (SUT): NFVI component
System Under Test (SUT): VNF
System Under Test (SUT): NFVI
System Under Test (SUT): VNF + NFVI
System Under Test (SUT): MANO
System Under Test (SUT): VNF + NFVI + MANO
Other Tools/Testing efforts
• CSIT
• SDN Tools:
• Cbench (OS) CBench is a somewhat classic SDN controller benchmark tool. It blasts a controller with OpenFlow packet-in messages and counts the rate of flow mod messages returned.
• WCBench: WCBench consumes CBench as a library, then builds a robust test automation, stats collection and stats analysis/graphing system around it.
• ClassBench-ng: Test utility for generation of firewall/OpenFlow rules based on original (no longer maintained) Classbench.
• NSTAT (OS): Goes beyond with Multi-Threaded Cbench, adds monitoring OF traffic, switch and controller scale testing.
• Veryx Technologies: Commercial tool, measures metrics in IETF Draft
• EXFO: Commercial tools, Load testing of IMS and EPC virtual network functions (VNF
• EEMBC creates their own tools for the benchmark specs they develop (Commercial)
• Rally
• ODL : OpenFlow protocol performance measurements in OpenDaylight.
• Canonical VNF Performance Interoperability Lab (V-PIL)Standards /
specificationsProjects Methodologies Tools Workloads