QoS, QoS Baby

OpenStack Barcelona 2016

2

Speakers

Anne McCormickSoftware Engineer, Cisco

@amccormi4

Robert StarmerCTO & Principal

Kumulus Technologies@rstarmer

Alka SathnurOps QA, Cisco

@alkasat12

3

• Traditional QoS Concepts

• Current/Future OpenStack QoS

• Beyond the Network

• Compute/Storage Bottlenecks and Differentiation

• Use Case

• Q/A

Topics

4

Traditional QoS Concepts

5

- Wikipedia

Quality of service (QoS) is the overall performance of a telephony or computer network, particularly the performance seen by the users of the network.

6

• Network centric view of resource• Availability• Reliability

• Provides a model for understanding and manipulating network impact on service delivery• Jitter/latency/loss all are important aspects

of a communications channel

What is Quality of Service Characteristic Description

Bandwidth Rate at which bits are moved across a network, often in raw bits/s

Latency Delay between sending a bit (source) and receiving it (destination)

Jitter Change in latency over time

Reliability Level of loss of bits across a network

7

“QoS” in OpenStack

Resource Gross Fine Name

Compute Reservation Resource share flavor

Network Rate limits Queueing qos

Storage Rate limits Queueing iopsStorage network Rate limits Queueing qos

8

• Initial QoS was managed by Routers• Commited Information Rate• Routers matched bandwidth between different networks

• Handling contention led to QoS Policies or Classes• Priority• Multi-queue

• And multiple models of handling those queues• FIFO• WRED

QoS in the “physical” Network

9

• L2 networks tend to try to avoid ever storing packets• Less chance to manage different flows of traffic

• But L2 networks really aren’t L2 any more• So we can classify traiffic and if necessary queue it• Really helps when you have multiple types of traffic like storage and

voice or video on the same network

QoS in Layer 2 Networks

10

Current/Future OpenStack QoS

11

• RXTX Factor• Nova network based “Sharing” algorithm• Based on nova flavor metadata

• Neutron Mitaka• ML 2 Extension• SR-IOV, OVS, Linux Bridge “bandwidth” limiations (e.g. rx/tx factor)

• Neutron Newton• As with Mitaka• Adds DSCP marking < This is a big deal

QoS in the early days

12

• Seems like a straight forward approach:• Like non-oversubscribed processors• Sharing fixed IOPs limits on a storage array

• Rate limiting flows or specific services can have unintended consequences:• Dramatic impact to “good put” vs. “through put”• Particularly bursty applications can become unstable

Rate Limiting

13

• Let’s help the network out• Mark packets so that the network infrastructure has some better

information to go on• Execute marking via application/OS level (VM

• or• Execute marking via the switch input

• Not a panacea• May still have “good put” impact• At least provides a better interaction for determining who gets access to

the available bandwidth resources

DSCP Marking

14

Beyond the Network

So you got the traffic there faster… now what?

Compute and storage bottlenecks!

15

Compute Bottlenecks

… and how to alleviate them

16

Controller

Compute1

Compute2

ComputeN

nova-scheduler

17

Controller

nova-scheduler

Node1 Node1

VeryImportantTM

VM CPU HogVeryImportantTM

VM

18

Controller

nova-scheduler

Node1 Node1

VeryImportantTM

VMVeryImportantTM

VM

19

Ran a simple OpenStack multicast iperf test:

• Network highly optimized for multicast (SR-IOV port, multiple rx queues with maximum queue size, RSS, ARFS, QoS)

• iPerf receiver on tenant VM, receiving steady 800 Mbits/sec multicast stream

• When context switching, receiver experienced up to 0.2% packet loss, particularly when switching across NUMA nodes (as opposed to switching within same node)

Cost of CPU Sharing/Context Switching

20

• Host aggregates – define separate groups of compute hosts

• Flavors – define hardware needs such as number of cores, CPU capabilities/limits, affinity/anti-affinity, etc., via host filters

• CPU pinning/NUMA awareness – pin VMs to dedicated cores to prevent context switches across NUMA nodes

Compute Resource Differentiation/Prioritization

21

Storage Bottlenecks

… and how to alleviate them

22

Storage2

StorageN

Compute

Compute

Compute

Compute

Storage1

VeryImportantTM

VM

I/O Traffic

I/O Hog

I/O Traffic

23

Ran a simple OpenStack read/write I/O test:

• Two VMs running on same host, different volumes

• 3 Ceph nodes, active/active/active

• When reading simultaneously, both VMs experienced 80 MB/s drop in read rate

• When writing simultaneously, both experienced 100 MB/s drop in write rate

Cost of Storage Contention

24

• Host aggregates – define separate groups/clusters of storage servers

• Flavors – define I/O bandwidth limits for VMs (outbound traffic)

• Differentiate at storage backend• Cinder has QoS specs, volume types, priority (more IOPS to particular volumes)• Ceph has storage types and the ability to limit IOPS to certain spindles using crush

maps• AFAIK, Swift does not have the ability to differentiate/prioritize storage resources at

the backend

Storage Resource Differentiation/Prioritization

25

• Network QoS is only a partial solution

• In order to guarantee resources for mission-critical applications and data, a solution across all cloud resources (network, compute, storage) must be used

• It is complicated to get this right across all resources, but it can be done

Conclusion

26

Use Case

27

Bringing an existing Content Delivery Network that is comprised of bare metal cache nodes onto an Openstack platform

Real World Use Case

28

Content Delivery Network• A Delivery Service is a software structure in OMD that maps an origin source to Traffic Servers

by the Fully Qualified Domain Name (FQDN). FQDN is in the Request URI from the client media player. Cache groups can belong to a single or multiple Delivery services

• Cache Group is a logical grouping for HA. Each cache is typically located in different location to provide site-level redundancy. Each cache in cache group associated has single geo coordinates.Enables:

Multiple Content Sources

Per Content Source content cache/storage

Intelligent load balancing

Origin Server

Traffic server (Edge)

Origin Server Origin Server

Traffic Server (Edge)

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Mid-tier cache

Edge cache Groups

Control Server

CDN Monitor

CDN Analytics

Orchestration

29

Content Delivery Network• A Delivery Service is a software structure in OMD that maps an origin source to Traffic Servers by

the Fully Qualified Domain Name (FQDN). FQDN is in the Request URI from the client media player. Cache groups can belong to a single or multiple Delivery services

• Cache Group is a logical grouping for HA. Each cache is typically located in different location to provide site-level redundancy. Each cache in cache group associated has single geo coordinates.

Origin Server

Traffic server (Edge)

Origin Server Origin Server

Traffic server (Edge)

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Traffic Server

Control Server

CDN Monitor

CDN Analytics

Director

Orchestration

Mid-tier cache

Edge Cache Groups and Storage Clusters

Traffic Server

Traffic Server

30

Dynamically expanding a Content Delivery Network is possible, provided the Orchestrator ensures that network, compute and storage give top priority to the application traffic. Thereby meeting the three goals – Performance, Availability and Capacity.

Use Case Summary

¿Preguntas?

32

¡Gracias!