Continuous Validation With SCTF

SDN Performance & Architecture EvaluationJason VennerMirantis

Vijay SeshadriCloud Platform Engineering (CPE), SymantecTODO:What does success look likeCPE Design principlesDC and network operations updateTL progress in the open source community Need to remove the symantec confidentialPublic presentation can't beCan't be confidential\AgendaCPE Overview1SDN Objectives and Test Plan2Test Setup and Framework3Test Results42Key Findings/Insights5 Introduction to CPECPE Overview CPE Charter Build a consolidated cloud infrastructure that offers platform services to host Symantec cloud applications Symantec cloud infrastructure already hosting diverse (security, data management) workloads Analytics Reputation based security, managed security services Storage Consumer and Enterprise backup/archival Network Hosted email & web security We are building an OpenStack based platform that provides additional storage and analytics services Secure multi-tenancy a core objective for all services- Given a sense of diversity workloads and scope of the platform. This not just a single application or OpenStack configured to be optimized for single app. - Out focus here is not just addressing scale related issues in various openstack environments, but to have the right framework for us to be able to identify and quickly remediate these issues. E.g SDN testing rapid creation of control plane elements VMs, networks, routers and tenant etc allowed us to identify scale issues in Neutron and fix some of them.- Architecture goals Secure, scalable and reliable OpenStack based cloud platform

Core ServicesCPE Platform Architecture

ComputeNetworkingStorageCLIs

Scripts

Cloud ApplicationsBig DataMessagingIdentity & Access (Keystone)Supporting ServicesAuthnRolesUser MgmtTenancyQuotasLoggingMeteringMonitoringDeploymentCompute (Nova)Image (Glance)SDN (Neutron)Load BalancingDNSSQLBatch AnalyticsStream ProcessingMsg QueueMem CacheEmail RelaySSLK/V StoreWeb Portal

Object Store

REST/JSON APIWe are following a RESTful SOA model- closely resembles OpenStack design philosophy- a collection of loosely coupled independent services that can be used to "compose" an application ( not an "all or nothing" proposition)- Keystone as a core IAC service, this means all other CPE services with integrate with keystone for Authn and Authz- All CPE services exposed thru a REST/JSON API.- Secure multi tenancy a core objective for all CPE Concept of domain and Tenants extended to hadoop, storm etc4CPE Overview1SDN Objectives and Test Plan2Test Setup and Framework3Test Results42Key Findings/Insights5 Introduction to CPESDN Objectives Provide secure multi-tenancy using strong network isolation Policy driven network access control within (and across) projects/domains Provide an ability for product engineering teams to define a network topology via REST APIsAssociate network objects dynamically with VMs, Projects (Tenants) Create and manage network access control policies within and across projects Enables easier integration of applications on partner infrastructure Interconnect OpenStack with bare metal storage/analytics services East-West traffic needs to transit the underlay network Support software driven network functions LBaaS, DNSaaS etcExpand on secure multi-tenancy Talk about the importance of cloud partners and CSBs and why automated provisioning and set up are important objectves----- Meeting Notes (4/15/14 11:28) -----1. Provisioning process workflow1. We need to customer provisioning workflow2. Test Plan Secure Multi Tenancy Test network isolation under various configurations Same Subnet Same network, different subnets Different networks, different tenants Different networks, identical IP addresses Test enforcement of network policies Deny All works and is the default Allow Specific works Removal of an Allow Specific works Data Plane Performance OpenStack Internal N x N Inter VM communication Client-Server TCP mesh using Iperf3Expand on secure multi-tenancy - ----- Meeting Notes (4/15/14 11:28) -----1. Provisioning process workflow1. We need to customer provisioning workflow2. Test Plan Contd Egress/Ingress Simulate data ingress/egress to non-OpenStack services (E.g Bare metal Hadoop cluster) Control Plane scalability Where does the solution break? Verify correct operation at the limit Test rate of creation and resource limits for neutron objects Number of Network Ports VNICs Number of Networks and Subnets Number of Routers Number of active flows Number of connections through the external Gateway (Ingress/Egress)Expand on secure multi-tenancy - ----- Meeting Notes (4/15/14 11:28) -----1. Provisioning process workflow1. We need to customer provisioning workflow2. CPE Overview1SDN Objectives and Test Plan2Test Setup and Framework3Test Results42Key Findings/Insights5 Introduction to CPEOverview of SDN solutions OpenStack is about networking OVS/VLAN 4k limit on VLANS Many VLANs spanning many TORs are operationally challenging, especially in rapidly changing environments No Failure Isolation Domain Overlay Simple Physical Network, each TOR L2 island L3 between the TORS Packet encapsulation for VM Traffic Controllers orchestrate tunnel mesh for VM trafficBuilding large L2 clouds, spanning many TORs is difficult, addin LACP in balance mode and you have many single points that can take down your whole cloud.In our first incarnation we had 100+ hosts, with 4000 Vlans, the STP cache was 3500 entries.

Large L2 is fragile, it can be done

Physical Test setup

140 2 socket servers7 Racks 80Gb to Spine2x10Gb LACP/Server2 Juniper MX Routers7 Racks of 20 servers 2x10G LACP per server, 1 TOR per rack 2x40 to the Spine, 1TB across the spine400GBit/sec within the TOR80 Gbit/sec Tor to SpineHypervisor 10 Gb/sGateway services 40 Gb/s

This is a nice small test lab. At this scale many of the underlying architectural limits start to bite hard. At half this size many issues will not surface.

Test Framework Overview We build a pool of VMs each placed into a specific Rack We have a control network we use to launch test cases and collect results

Use OpenStack to Test OpenStackWe ended up using ZeroMQ.The test tool, actually provisioned the virtual resources for a given test.Orchestrating 1000s of VMs and hundreds of subnets and routers is fun, for some definitions of funNSX Test Setup: OpenStack Internal

4NSX Controllers4NSX Gateways3 OpenSackControllersBottom RightNSX ControllersNSX Gateway servers

NSX ServicesWe took 8 servers and used them for NSX4 controllers4 gateway servers Broadcast,Multicast etcNSX Test Setup: Ingress/Egress

4NSX Controllers20

NSX

Gateways20

Outsiide hosts

Traffic SourceSink3 OpenSackControllerNot HypervisorsWhen we did the ingress/egress we took an entire rack and used those servers as gatewaysWe carved out another rack to use as our external hosts.Contrail Test Setup: OpenStack Internal

4 servers2x10 Used3 serversConrollers1 OpenSackControllerStole the TOR ports for the MXsIdeally the Juniper MXs would have been hung off our spine.What we ended up with was them sitting in the rack we used for the NSX gateway services, and we took the 2x10G ports out of the last 4 servers to give us 80GB into the MXs

Contrail Test Setup: Ingress/Egress

4 servers2x10 Ports Used20

OutSide

TrafficSourceSink1 OpenSackControllerHypervisorsWe could use the rest of the servers in the rack for hypervisorsCost of Gear is handled by another team

CPE Overview1SDN Objectives and Test Plan2Test Setup and Framework3Test Results42Key Findings/Insights5 Introduction to CPEMulti-tenancy Test Results Connectivity within a subnet Both solutions proved isolation Contrail 1.04 was not deny all by default Same network, different subnets Both solutions proved isolation

We ran these by hand.The first packet might see some additional lantecyMulti-tenancy Test Results Different networks in different projects Both solutions proved isolation Contrail used floating IPs Different network overlapping IP addresses Both solutions proved isolation

Data Plane Test Setup

2000 VMs take a while to launchOur assumption is the default is to deny all, and allow only what is explicitly defined.VMs on same subnet, in same networkVMs on separate subnets in same network, with routerVMs on separate subnets, in separate networks, traversing external networkVMs with identical IP addresses, using floating IP addresses to communicateCIDR based security groups and named security groupsVerify deny all behavior, add allow rule, verify allowed, delete allow, verify deny allData Plane Test Results: OpenStack Internal Both overlay solutions added 4 % payload overhead This is in addition to 3% underlay frame overhead Both solutions ran close to virtual wire speed. We hit 75Gb/sec per TOR out of 80Gb/sec Peak across the TORS 225Gb/sec out of 240Gb/sec Traversing SDN routers had little impact on peak performance Neutron OVS/VLAN required jumbo frames to hit wire speedAdd more details around the tests, and set of metrics we collected, consider displaying it as a table.

Data Plane Test Results: Ingress/EgressOn a Per VM Basis we ran at virtual wire speed to/from our external hostsOVS bugs in OVS 2.1 limited the performance of individual connections under high concurrency (>10,000)Saturation of our TOR to Spine connection was the principle performance limitGateway traffic required 2 passes through our Gateway TOR one busy TORSaturated the Gateway TOR, data payload at 37 Gb/secTested up to 100,000 concurrent connectionsOVS performance fell off after 10,000vRouter was able to scale till 100,000

The TOR on the rack providing the gateway services was saturated by our testsOur limit was the physical network architecture, not the SDN solutionControl Plane Test ResultsTestNSX QuantityNSX NotesContrail QuantityContrail NotesNumber of Networks>16,000Limiting factor is the amount of RAM in the NSX controllers>2,500OpenStack configuration limited the testNumber of Subnets per Network>1024OpenStack configuration limited the test

>1024OpenStack configuration limited the testNumber of Network Ports>8192OpenStack configuration limited the test>5000Hardware Limits in the cluster limited the testNumber of Networks per Router> 512Published limit is 10>300OpenStack configuration limited the test

CPE Overview1SDN Objectives and Test Plan2Test Setup and Framework3Test Results42Key Findings/Insights5 Introduction to CPEKey Findings: OpenStack configuration Building a 100+ node OpenStack cluster requires extensive config tuning Host OS, MySQL, DB Indexes and Rabbit tuning Horizontal scaling of API servers API service thread counts, connection pools and queues Python REST servers are not the ideal for handling 10,000s of requests per second Keystone configuration Memory cache for tokens Short lived tokens with regular purge Havana Neutron implementation does not meet our scalability goals 8 minutes+ to turn up a network port, on a network with a few hundred VMs Long delays with the query APIs when there are a few thousand VMs

Key findings: VM Network Performance Tune your VM kernels With the default Ubuntu 12.04 or CentOS 6.5 kernel settings for network buffer space and TCP window sizes we see 1.8Gb/sec/VM with 1500byte MTU. With standard 10G tuning for rmem/wmem and interface txqueuelen

MTU 1500MTU 9000OVS/VLAN3.8Gb/sec/VM9Gb/sec/VMNSX9.1Gb/sec/VMContrail9.1Gb/sec/VM# increase TCP max buffer size settable using setsockopt()net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 # increase Linux autotuning TCP buffer limit net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 65536 16777216# increase the length of the processor input queuenet.core.netdev_max_backlog = 30000# recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp# recommended for hosts with jumbo frames enablednet.ipv4.tcp_mtu_probing=1Conclusion SDN is a core capability for us to offer a secure multi-tenant cloud platform Overlay solutions provide a strong network isolation and access control Our use case requires extensive traffic into and out of the SDN zone NSX requires host servers and additional network configuration Contrail uses MPLS enabled routers, more integrated with underlay infrastructure Both overlay solutions met our short term performance and scalability goals We will continue to evaluate the SDN space for solutions that meet our long term goals

Thank you!SYMANTEC Copyright 2014 Symantec Corporation. All rights reserved.28