RALLY – PERFORMANCE

BENCHMARKING OF

OPENSTACK

Tips and tools for creating and presenting wide format slides

Openstack – cloud platform

Current Openstack - Ecosysten

OpenStack Basics

Everything is written in python

End users can interact through a common web interface (Horizon) or directly to each service through their API

All services authenticate through a common source

Individual services interact with each other through their public APIs *

Most daemons implemented WSGI middleware (Paste)

Used extensively in OpenStack

Configured through *-paste.ini files

4

Identity (“Keystone”)

Keystone provides a single point of integration for OpenStack policy, catalog, token and authentication.

keystone handles API requests as well as providing configurable catalog, policy, token and identity services.

Standard backends include LDAP or SQL, as well as Key Value Stores (KVS).

Most people will use this as a point of customization for their current authentication services.

OpenStack Identity Service

keystone(ser vice & admin APIs)

ident it y backend

catalog backend

token backend policy backend

5

Dashboard (“Horizon”)

Django application that users can access in their web browser

Communicates with each OpenStack service through their API (and sometimes their admin API)

OpenStackDashboard

Hor izon

HTTP(S)

6

Object Storage (“Swift”)

OpenStack Object Store

swif t-proxy

objectcontaineraccount

objectstore

accountDB

containerDB

memcached

Stores and serves objects (files)

Employs object level replication to safeguard data

Accepts client requests via Objectstore API or HTTP from clients through swift-proxy

Maintains distributed account and container databases

Stores objects according the ring layout on filesystem with extended attributes (XFS, EXT4, etc.)

7

Image Service (“Glance”)

glance-api accepts Image API calls for image discovery, image retrieval and image storage.

glance-registry stores, processes and retrieves metadata about images (size, type, etc.).

Database to store the image metadata.

A storage repository for the actual image files. In many deployments, this is OpenStack Swift

OpenStack Image Ser vice

glance-api

glance-regist r y

glancedatabase

8

Compute (“Nova”)

OpenStack Compute

nova-api(OS, EC2, Admin)

nova-consoleauth

nova-cert/objectstore

nova-console

Queue

nova-compute

nova-scheduler

novadatabase

hyper visor

libvirt, XenAPI, etc.

nova-conductor

nova-api accepts and responds to end user compute API calls.

Supports OpenStack Compute API, Amazon's EC2 API and a special Admin API (for privileged users to perform administrative actions).

Initiates most of the orchestration activities (such as running an instance)

Enforces some policy (mostly quota checks)

Authentication is handled through middleware before getting to this daemon

9

Block Storage (“Cinder”) cinder-api accepts API requests and

routes them to cinder-volume for

action.

cinder-volume acts upon the

requests by reading or writing to the

Cinder database to maintain state,

interacting with other processes (like

cinder-scheduler) through a message

queue and directly upon block

storage providing hardware or

software. It can interact with a

variety of storage providers through a

driver architecture. Currently, there

are drivers for IBM, SolidFire, NetApp,

Nexenta, Zadara, linux iSCSI and

other storage providers.

Much like nova-scheduler, the

cinder-scheduler daemon picks the

optimal block storage provider node

to create the volume on.

OpenStack Block Storage

cinder-api

cinder-volume

cinder-scheduler

cinderdatabase

volume provider

10

Networking (“Quantum”)

OpenStack Net work Ser vice

quant um-ser ver

quant um plugin(s)

net workprovider

Queue

quant umdatabase

quant um agent (s)

quantum-server accepts API requests and then routes them to the appropriate quantum plugin for action.

Quantum ships with plugins and agents for:

Cisco virtual and physical switches

Nicira NVP product

NEC OpenFlow products

Open vSwitch

Linux bridging

Ryu Network Operating System

Midokua

The common agents are L3 (layer 3), DHCP (dynamic host IP addressing) and the specific plug-in agent.

11

Rally - Why ?

"How does OpenStack work at scale?”

Rally Components

Server Providers - provide servers (virtual servers), with ssh access, in one L3

network.

Deploy Engines - deploy OpenStack cloud on servers that are presented by

Server Providers

Verification - component that runs tempest (or another specific set of tests)

against a deployed cloud, collects results & presents them in human readable

form.

Benchmark engine - allows to write parameterized benchmark scenarios & run

them against the cloud.

Rally -Functionality

Rally Use Cases

Installing Rally on Ubuntu.

Prerequisite

• sudo apt-get update

• sudo apt-get install libpq-dev git-core python-dev libevent-dev libssl-dev python-pip libffi-dev

• sudo pip install pbr

Note : We would recommend using a separate machine for Rally. If the machine that

you use to run rally has some Openstack components running, we suggest creating a

virtual environment for running Rally because it may have conflicts with version of the

client python libraries.

Installing Rally on Ubuntu. Installing Rally

Clone

• git clone https://github.com/stackforge/rally.git && cd rally

• python setup.py install

Configure

• sudo mkdir /etc/rally

• sudo cp etc/rally/rally.conf.sample /etc/rally/rally.conf

• sudo vim /etc/rally/rally.conf

• # Change the "connection" parameter,

• # e.g. to connection="sqlite://///home/<your_username>/.venv/rally-db/$sqlite_db" (or any other place)

Create Database

• rally-manage db recreate

Rally : Deploy Engines.

The task of a deploy engine is to control the process of deploying some OpenStack

distribution like DevStack or FUEL before any benchmarking procedures take place.

Every deploy engine should implement the following fairly simple interface:

• constuctor, which takes a deployment entity as its only parameter;

• deploy(), which should deploy the appropriate OpenStack distribution given the cloud

config from the deployment object the engine was initialized with (possibly using one

of available server providers)

• cleanup(), which should clean up the OpenStack deployment (again, possibly using one

of available server providers).

Rally. Deploy Engine . Example If you already have a existing Openstack Deployment that you want to benchmark

Use a json file that looks like one below. (with value specific to your system):

{

"name": "DummyEngine",

"endpoint": {

"auth_url": "http://192.168.122.22:5000/v2.0/",

"username": "admin",

"password": "password",

"tenant_name": "admin"

}

}

For Devstack based deployment {

"name": "DevstackEngine",

"localrc": {

"ADMIN_PASSWORD": "secret",

"NOVA_REPO": "git://example.com/nova/",

...

},

"devstack_repo": "git://example.com/devstack/",

"provider": {

"name": "${PROVIDER_NAME}",

...

}

}

Rally – Server Providers Server providers in Rally are typically used by deploy engines to manage virtual

machines necessary for OpenStack deployment and its following benchmarking.

The key feature of server providers is that they provide a unified interface for

interacting with different virtualization technologies (LXS, Virsh etc.) and cloud

suppliers (like Amazon).

Every server provider should implement the following basic interface:

• constructor, which should take the deployment entity the provider should bind to and a

config dictionary as its parameters;

• create_servers(image_uuid, type_id, amount), which should create the requested number of

virtual machines of the given type using a specific image. The method should also return the

list of created servers wrapped in special Server entities.

• destroy_servers(), which should destroy all virtual machines previously created by the same

server provider.

Rally – Server Providers –Ex DummyProvider – If you already have an Openstack Deployment.

This provider does nothing, but returns endpoints from configuration. This may be

useful if you have specific software/hardware configuration ready to deploy

OpenStack.

{

"name": "ExampleEngine",

"provider": {

"name": "DummyProvider",

"credentials": [{"user": "root", "host": "host1.net"}, {"user": "root", "host": "host2.net"}]

}

}

Rally – How_to_run

Simple

1. Initialize your Deployment

2. Create a json for Supported Benchmarking scenario

3. Run Benchmarking against a deployment above

Rally – Initialize Deployment 1. Create a Deployment configuration (json) file.

If you are running Rally against a existing Openstack Deployment your should look like

{

"name": "DummyEngine",

"endpoint": {

"auth_url": <KEYSTONE_AUTH_URL>,

"username": <ADMIN_USER_NAME>,

"password": <ADMIN_PASSWORD>,

"tenant_name": <ADMIN_TENANT>

}

}

2. Create a deployment using deployment create command

$ rally deployment create --filename=dummy_deployment.json --name=dummy

3. If you want to list deployments

$ rally deployment list

4. Switch to a different Deployment

$ rally use deployment --deploy-id=<Another deployment UUID>

Rally – Set Benchmark scenario. Some sample configurations are provided at “rally/doc/samples/tasks/”.

Lets pick up a scenario boot-and-delete.json from Nova. It looks like

{

"NovaServers.boot_and_delete_server": [

{

"args": {"flavor_id": 1,

"image_id": "73257560-c59b-4275-a1ec-ab140e5b9979"},

"execution": "continuous",

"config": {"times": 10, "active_users": 2, "tenants": 3,

"users_per_tenant": 2}

}

]

}

Modify this to design your test-case. Similarly for all other cases other available json can be modified or you can even write a new one for a custom case.

Rally by default uses the last created deployment. Use the switch deployment commands to run the tests with a different deployment.

Lets dig deeper

Test name : “NovaServers.boot_and_delete_server”

Execution : either continuous / periodic

Times : Number of times the test needs to be run

Active_users : Number of parallel threads (concurrent users)

Tenants : Total number of tenants to be created

Users_per_tenant : Number of users per single tenant

Other parameters to be used only with supported tests

Network : Name of network to be used

Script : If a script is passed as input to the test

Actions : soft_reboot / stop_start

Rally – Run Benchmark. Run your benchmark scenario by pointing at the json you created in the previous step

$ rally --verbose task start --task=my-task.json

You can check the state of the task

$ rally task list

To check a complete task analysis

$ rally task detailed <Task UUID>

Rally Result –Example.

$ rally task detailed <Task UUID>

===========================================================================

=====

Task <Task UUID> is finished. Failed: False

--------------------------------------------------------------------------------

test scenario NovaServers.boot_and_delete_server

args position 0

args values:

{u'args': {u'flavor_id': <Flavor UUID>,

u'image_id': u'<Image UUID>'},

u'config': {u'active_users': 1, u'times': 2}}

+--------------------+---------------+---------------+---------------+

| action | max (sec) | avg (sec) | min (sec) |

+--------------------+---------------+---------------+---------------+

| nova.boot_server | 9.22798299789 | 8.90022659302 | 8.57247018814 |

| nova.delete_server | 4.24928498268 | 3.26377093792 | 2.27825689316 |

+--------------------+---------------+---------------+---------------+

+---------------+---------------+---------------+---------------+-------------+

| max (sec) | avg (sec) | min (sec) | success/total | total times |

+---------------+---------------+---------------+---------------+-------------+

| 13.4775559902 | 12.1641695499 | 10.8507831097 | 1.0 | 2 |

+---------------+---------------+---------------+---------------+-------------+

--------------------------------------------------------------------------------

Detailed performance benchmarking

Docker in OpenStack • Havana

– Nova virt driver which integrates with docker REST API on backend – Glance translator to integrate docker images with Glance

• Icehouse – Heat plugin for docker

• Both options are still under development

9/12/2014 29

nova-docker virt driver docker heat plugin

About This Benchmark • Use case perspective

– As an OpenStack Cloud user I want a Ubuntu based VM with MySQL… Why would I choose docker LXC vs a traditional hypervisor?

• OpenStack “Cloudy” perspective – LXC vs. traditional VM from a Cloudy (OpenStack) perspective – VM operational times (boot, start, stop, snapshot) – Compute node resource usage (per VM penalty); density factor

• Guest runtime perspective – CPU, memory, file I/O, MySQL OLTP

• Why KVM? – Exceptional performance

9/12/2014 30

Benchmark Environment Topology @ SoftLayer

glance api / reg nova api / cond / etc

keystone …

rally

nova api / cond / etc cinder api / sch / vol

docker lxc

dstat

controller compute node

glance api / reg nova api / cond / etc

keystone …

rally

nova api / cond / etc cinder api / sch / vol

KVM

dstat

controller compute node

9/12/2014 31

+ Awesome!

+ Awesome!

Benchmark Specs

9/12/2014 32

Spec Controller Node (4CPU x 8G RAM) Compute Node (16CPU x 96G RAM)

Environment Bare Metal @ SoftLayer Bare Metal @ SoftLayer

Mother Board SuperMicro X8SIE-F Intel Xeon QuadCore SingleProc SATA [1Proc] SuperMicro X8DTU-F_R2 Intel Xeon HexCore DualProc [2Proc]

CPU Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] (Intel Xeon-Westmere 5620-Quadcore [2.4GHz]) x 2

Memory (Kingston 4GB DDR3 2Rx8 4GB DDR3 2Rx8 [4GB]) x2 (Kingston 16GB DDR3 2Rx4 16GB DDR3 2Rx4 [16GB]) x 6

HDD (LOCAL) Digital WD Caviar RE3 WD5002ABYS [500GB]; SATAII Western Digital WD Caviar RE4 WD5003ABYX [500GB]; SATAII

NIC eth0/eth1 @ 100 Mbps eth0/eth1 @100 Mbps

Operating System Ubuntu 12.04 LTS 64bit Ubuntu 12.04 LTS 64bit

Kernel 3.5.0-48-generic 3.8.0-38-generic

IO Scheduler deadline deadline

Hypervisor tested NA - KVM 1.0 + virtio + KSM (memory deduplication) - docker 0.10.0 + go1.2.1 + commit dc9c28f + AUFS

OpenStack Trunk master via devstack Trunk master via devstack. Libvirt KVM nova driver / nova-docker virt driver

OpenStack Benchmark Client OpenStack project rally NA

Metrics Collection NA dstat

Guest Benchmark Driver NA - Sysbench 0.4.12 - mbw 1.1.1.-2 - iibench (py) - netperf 2.5.0-1

VM Image NA - Scenario 1 (KVM): official ubuntu 12.04 image + mysql snapshotted and exported to qcow2 – 1080 MB

- Scenario 2 (docker): guillermo/mysql -- 334.8 MB

Test Descriptions: Cloudy Benchmarks

9/12/2014 33

Benchmark Benchmark Driver Description

OpenStack Cloudy Benchmarks

Serial VM boot (15 VMs)

OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Repeat the above a total of 15 times - Delete VMs

VM reboot (5 VMs rebooted 5 times each)

OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Soft reboot VM 5 times - Delete VM - Repeat the above a total of 5 times

VM snapshot (1 VM, 1 snapshot)

OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Snapshot VM to glance image - Delete VM

Test Descriptions: Guest Benchmarks

9/12/2014 34

Benchmark Benchmark Driver Description

Guest Runtime Benchmarks

CPU performance Sysbench from within the guest - Run sysbench cpu test - Repeat a total of 10 times - Average results over the 10 times

OLTP (MySQL) performance Sysbench from within the guest - Run sysbench OLTP test - Repeat a total of 10 times - Average results over the 10 times

MySQL Indexed insertion benchmark - Run iibench for a total of 1M inserts printing stats at 100K intervals - Collect data over 5 runs & average

File I/O performance Sysbench from within the guest - Synchronous IO - No-direct

- Run sysbench OLTP test - Repeat a total of 10 times - Average results over the 10 times

Memory performance Mbw from within the guest - Run mbw with array size of 1000 MiB and each test 10 times - Collect average over 10 runs per test

Network performance Netperf - Run netperf server on controller - From guest run netperf client in IPv4 mode - Repeat text 5x - Average results

SERIALLY BOOT 15 VMS OpenStack Cloudy Benchmark

9/12/2014 35

Cloudy Performance: Serial VM Boot

• Benchmark scenario overview – Boot VM via OpenStack nova – Wait for VM to become active – Repeat the above steps for a total of 15 VMs – Delete all VMs

9/12/2014 36

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Act

ive

VM

s

Time

Benchmark Visualization

VMs

Cloudy Performance: Serial VM Boot

9/12/2014 37

3.900927941

5.884197426

0

1

2

3

4

5

6

7

docker KVM

Tim

e in

Se

con

ds

Average Server Boot Time

docker

KVM

Docker 1.5x faster

Cloudy Performance: Serial VM Boot

9/12/2014 38

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89CP

U U

sage

In

Pe

rce

nt

Time

Docker: Compute Node CPU

usr

sys

0

20

40

60

80

100

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125129133137CP

U U

sage

In

Pe

rce

nt

Time

KVM: Compute Node CPU

usr

sys

Averages

– 1.14

– 0.44

Averages

– 12.6

– 2.08

Cloudy Performance: Serial VM Boot

9/12/2014 39

0.00E+00

1.00E+09

2.00E+09

3.00E+09

4.00E+09

5.00E+09

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89

Me

mo

ry U

sed

Time

Docker: Compute Node Used Memory

Memory

0.00E+00

1.00E+09

2.00E+09

3.00E+09

4.00E+09

5.00E+09

1 5 9

13

17

21

25

29

33

37

41

45

49

53

57

61

65

69

73

77

81

85

89

93

97

10

1

10

5

10

9

11

3

11

7

12

1

12

5

12

9

13

3

13

7

Me

mo

ry U

sed

Time

KVM: Compute Node Used Memory

Memory

Delta 687 MB Per VM 45.8 MB

Delta 2775 MB Per VM 185 MB

Cloudy Performance: Serial VM Boot

9/12/2014 40

0

5

10

15

20

25

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89

1 M

inu

te L

oad

Ave

rage

Time

Docker: Compute Node 1m Load Average

1m

0

5

10

15

20

25

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125129133137

1 M

inu

te L

oad

Ave

rage

Time

KVM: Compute Node 1m Load Average

1m

Average 0.09 %

Average 9.94 %

SERIAL VM SOFT REBOOT OpenStack Cloudy Benchmark

9/12/2014 41

Cloudy Performance: Serial VM Reboot • Benchmark scenario overview

– Boot a VM

• Wait for it to become active

– Soft reboot the VM

• Wait for it to become active

• Repeat soft reboot a total of 5 times

– Delete VM

– Repeat the above for a total of 5 VMs

9/12/2014 42

0

1

2

3

4

5

6

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55

Act

ive

VM

s

Time

Benchmark Visualization

Active VMs

Cloudy Performance: Serial VM Reboot

9/12/2014 43

6.591313448

124.4525079

0

20

40

60

80

100

120

140

docker KVM

Tim

e In

Se

con

ds

Average Server Reboot Time

docker

KVM

Docker 18.9x faster

Cloudy Performance: Serial VM Reboot

9/12/2014 44

7.857602167

4.451871514

0

1

2

3

4

5

6

7

8

9

docker KVM

Tim

e In

Se

con

ds

Average Server Delete Time

docker

KVM

KVM 1.75x faster

Cloudy Performance: Serial VM Reboot

• After investigating docker delete times – Docker sends SIGTERM to container process to stop

– Bash is immune to SIGTERM

– Docker waits for X seconds before stop

– Default container image (init) command was using bash

– See: https://github.com/dotcloud/docker/issues/3766

• Rebuild the docker mysql image – Don’t use bash for container image command

– No change to docker (hypervisor); image config only change

• Rerun rally reboot tests; Cloudy reboot round 2

9/12/2014 45

Cloudy Performance: Serial VM Reboot

9/12/2014 46

2.541945305

124.4525079

0

20

40

60

80

100

120

140

docker KVM

Tim

e In

Se

con

ds

Average Server Reboot Time (Round 2)

docker

KVM

Docker 48.99x faster

Cloudy Performance: Serial VM Reboot

9/12/2014 47

4.093254519

4.451871514

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

docker KVM

Tim

e In

Se

con

ds

Average Server Delete Time (Round 2)

docker

KVM

Docker 1.09x faster

Cloudy Performance: Serial VM Reboot

9/12/2014 48

0

2

4

6

8

10

1

74

14

7

22

0

29

3

36

6

43

9

51

2

58

5

65

8

73

1

80

4

87

7

95

0

10

23

10

96

11

69

12

42

13

15

13

88

14

61

15

34

16

07

16

80

17

53

18

26

18

99

19

72

20

45

21

18

21

91

22

64

23

37

24

10

24

83

25

56

26

29

27

02

27

75

28

48

29

21

29

94

30

67

31

40

CP

U U

sage

In

Pe

rce

nt

Time

KVM: Compute Node CPU

usr

sys

0

2

4

6

8

101 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

10

3

10

9

11

5

12

1

12

7

13

3

13

9

14

5

15

1

15

7

16

3

16

9

17

5

18

1

18

7

19

3

19

9

20

5

21

1

21

7

22

3

22

9

23

5

24

1

CP

U U

sage

In

Pe

rce

nt

Time

Docker: Compute Node CPU

usr

sys

Averages

– 0.34

– 0.11

Averages

– 0.82

– 0.19

Cloudy Performance: Serial VM Reboot

9/12/2014 49

0.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

1 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

10

3

10

9

11

5

12

1

12

7

13

3

13

9

14

5

15

1

15

7

16

3

16

9

17

5

18

1

18

7

19

3

19

9

20

5

21

1

21

7

22

3

22

9

23

5

24

1

Me

mo

ry U

sed

Time

Docker: Compute Node Used Memory

Memory

0.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

17

91

57

23

53

13

39

14

69

54

76

25

70

37

81

85

99

37

10

15

10

93

11

71

12

49

13

27

14

05

14

83

15

61

16

39

17

17

17

95

18

73

19

51

20

29

21

07

21

85

22

63

23

41

24

19

24

97

25

75

26

53

27

31

28

09

28

87

29

65

30

43

31

21

Me

mo

ry U

sed

Time

KVM: Compute Node Used Memory

Memory

Delta 57 MB

Delta 467 MB

Cloudy Performance: Serial VM Reboot

9/12/2014 50

0

1

2

3

4

17

11

41

21

12

81

35

14

21

49

15

61

63

17

01

77

18

41

91

19

81

10

51

11

21

11

91

12

61

13

31

14

01

14

71

15

41

16

11

16

81

17

51

18

21

18

91

19

61

20

31

21

01

21

71

22

41

23

11

23

81

24

51

25

21

25

91

26

61

27

31

28

01

28

71

29

41

30

11

30

81

31

51

1 M

inu

te L

oad

Ave

rage

Time

KVM: Compute Node 1m Load Average

1m

-0.5

0.5

1.5

2.5

3.51 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

10

3

10

9

11

5

12

1

12

7

13

3

13

9

14

5

15

1

15

7

16

3

16

9

17

5

18

1

18

7

19

3

19

9

20

5

21

1

21

7

22

3

22

9

23

5

24

1

1 M

inu

te L

oad

Ave

rage

Time

Docker: Compute Node 1m Load Average

1m

Average 0.05 %

Average 0.35 %

SNAPSHOT VM TO IMAGE OpenStack Cloudy Benchmark

9/12/2014 51

Cloudy Performance: Snapshot VM To Image

• Benchmark scenario overview

– Boot a VM

– Wait for it to become active

– Snapshot the VM

– Wait for image to become active

– Delete VM

9/12/2014 52

Cloudy Performance: Snapshot VM To Image

9/12/2014 53

26.39477992

42.92771101

0

5

10

15

20

25

30

35

40

45

50

docker KVM

Tim

e (

Seco

nd

s)

Average Snapshot Server Time

docker

KVM

Docker 1.62x faster

Cloudy Performance: Snapshot VM To Image

9/12/2014 54

0

2

4

6

8

1 4 7

10

13

16

19

22

25

28

31

34

37

40

43

46

49

52

55

58

61

64

67

70

73

76

79

82

85

88

91

94

97

10

0

10

3

10

6

10

9

11

2

CP

U U

sage

In

Pe

rce

nt

Time

KVM: Compute Node CPU

usr

sys

-1

1

3

5

7

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67CP

U U

sage

In

Pe

rce

nt

Time

Docker: Compute Node CPU

usr

sys

Averages

– 0.4

– 0.11

Averages

– 1.58

– 1.07

Cloudy Performance: Snapshot VM To Image

9/12/2014 55

1.65E+09

1.7E+09

1.75E+09

1.8E+09

1.85E+09

1 4 7

10

13

16

19

22

25

28

31

34

37

40

43

46

49

52

55

58

61

64

67

70

73

76

79

82

85

88

91

94

97

10

0

10

3

10

6

10

9

11

2

Me

mo

ry U

sed

Time

KVM: Compute Node Used Memory

Memory

1.82E+091.84E+091.86E+091.88E+09

1.9E+091.92E+091.94E+09

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67

Me

mo

ry U

sed

Time

Docker: Compute Node Used Memory

Memory

Delta 48 MB

Delta 114 MB

Cloudy Performance: Snapshot VM To Image

9/12/2014 56

0

0.2

0.4

0.6

1 4 7

10

13

16

19

22

25

28

31

34

37

40

43

46

49

52

55

58

61

64

67

70

73

76

79

82

85

88

91

94

97

10

0

10

3

10

6

10

9

11

2

1 M

inu

te L

oad

Ave

rage

Time

KVM: Compute Node 1m Load Average

1m

0

0.2

0.4

0.6

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67

1 M

inu

te L

oad

Ave

rage

Time

Docker: Compute Node 1m Load Average

1m

Average 0.1 %

Average 0.37 %

GUEST PERFORMANCE BENCHMARKS

Guest VM Benchmark

9/12/2014 57

Guest Performance: CPU

• Linux sysbench 0.4.12 cpu test

• Calculate prime numbers up to 20000

• 2 threads

• Instance size

– 4G RAM

– 2 CPU cores

– 20G disk 9/12/2014 58

Guest Performance: CPU

9/12/2014 59

15.11 15.08 15.03

0

2

4

6

8

10

12

14

16

Docker KVM Bare Metal

Seconds

Calculate Primes

Docker

KVM

Bare Metal

Guest Performance: Memory

• Linux mbw 1.1.1-2

• Instance size

– 2 CPU

– 4G memory

• Execution options

– 10 runs; average

– 1000 MiB 9/12/2014 60

Guest Performance: Memory

9/12/2014 61

0

2000

4000

6000

8000

10000

12000

14000

MEMCPY DUMB MCBLOCK

MiB

/s

Memory Tests

Memory Benchmark Performance

BareMetal

docker

KVM

Guest Performance: Network

• Netperf 2.5.0-1 – Netserver running on controller

– Netperf on guest

– Run netperf 5 times & average results

• Instance size – 2 CPU

– 4G memory

• Execution options – IPv4 / TCP

9/12/2014 62

Guest Performance: Network

9/12/2014 63

0

100

200

300

400

500

600

700

800

900

1000

docker KVM

Thro

ugh

pu

t In

10

^6 b

its/

seco

nd

Network Throughput

docker

KVM

Guest Performance: File I/O • Linux sysbench 0.4.12 fileio test

– Synchronous IO – Random read / write – Total file size of 150G – 16K block size

• Thread variations: 1, 8, 16, 32 • Instance size

– 4G RAM – 2 CPU cores – 200G disk

• KVM specs – Disk cache set to none – Virtio – Deadline scheduler (host & guest)

• Docker specs – AUFS storage driver – Deadline scheduler

9/12/2014 64

Guest Performance: File I/O

9/12/2014 65

0

100

200

300

1 8 16 32

Mb

Threads

File I/O: Read

Docker

KVM

0

50

100

150

200

1 8 16 32

Mb

Threads

File I/O: Write

Docker

KVM

Guest Performance: File I/O – Read / Write

9/12/2014 66

0

200

400

600

800

1000

1200

1400

1600

1 8 16 32

Kb

/se

c

Threads

File I/O: Transfer Rate

Docker

KVM

Guest Performance: MySQL OLTP • Linux sysbench 0.4.12 oltp test

– Table size of 2,000,000 – MySQL 5.5 (installed on Ubuntu 12.04 LTS with apt-get)

• Variations – Number of threads – Read only & read / write

• Instance size – 4G RAM – 2 CPU cores – 20G disk

9/12/2014 67

Guest Performance: MySQL OLTP (Read)

9/12/2014 68

0

20000

40000

60000

2 8 16

Tota

l Tra

nsa

ctio

ns

Threads

MySQL OLTP Read Transactions

Docker

KVM

0

200000

400000

600000

800000

2 8 16

Re

qu

est

s

Threads

MySQL OLTP Read Requests

Docker

KVM

Guest Performance: MySQL OLTP (Read / Write)

9/12/2014 69

0

5000

10000

15000

20000

25000

2 16 32 64

Tota

l Tra

nsa

ctio

ns

Threads

MySQL OLTP Read/Write Transactions

docker

KVM

0

100000

200000

300000

400000

2 16 32 64

Re

qu

est

s

Threads

MySQL OLTP Read/Write Requests

Docker

KVM

Guest Performance: MySQL Indexed Insertion

• Indexed insertion benchmark (iibench python script) – A total of 1,000,000 insertions

– Print stats at 100K intervals

– Collect stats over 5 runs

– Average

• Instance size – 4G RAM

– 2 CPU cores

– 20G disk

9/12/2014 70

Guest Performance: MySQL Indexed Insertion

9/12/2014 71

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10

Seco

nd

s P

er

10

0K

In

sert

ion

Bat

ch

Table Size In 100K Increments

MySQL Indexed Insertion @ 100K Intervals

docker

kvm

BENCHMARK OBSERVATIONS

9/12/2014 72

Docker LXC CPU Growth 26x Lower Than VM

9/12/2014 73

y = 0.0091x + 0.7349

y = 0.237x + 2.2993

0

5

10

15

20

25

30

351 4 7

10

13

16

19

22

25

28

31

34

37

40

43

46

49

52

55

58

61

64

67

70

73

76

79

82

85

88

91

94

97

10

0

10

3

CP

U U

sage

In

Pe

rce

nt

User CPU Growth Trend

docker

KVM

Linear (docker)

Linear (KVM)

Slope

– 0.0091

– 0.237

Docker LXC Memory Growth 3x Lower Than VM

9/12/2014 74

y = 1E+07x + 1E+09

y = 3E+07x + 1E+09

0.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

3.00E+09

3.50E+09

4.00E+09

4.50E+09

5.00E+09

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94

Me

mo

ry U

sed

Memory Usage Growth Trend

docker

KVM

Linear (docker)

Linear (KVM)

Slope

– 1E+07

– 3E+07

Observations • Cloudy operations with Docker LXC outperform VM

– 48x server reboot, 1.5x server boot, 1.62x server snapshot, etc.

• Docker LXC density potential compared to VMs – 3x memory savings – 26x CPU savings – 3.22x smaller images in this test (note – image sizes can vary based on required packages)

• Docker LXC containers run on bare metal – Greater / equivalent “in the VM” performance at the micro benchmark level

• Micro benchmark results do not always reflect macro performance – Always benchmark your “real” workload

• Docker image (init) command can impact performance – Bash ignores SIGTERM

• Nova-docker virt driver and docker-registry components still under dev – Nice work, but room for improvement (python anyone?)

• Real performance of Docker LXC ops capped my Cloud manager – Can start the SQL image from docker CLI in 0.191s

9/12/2014 75

REFERENCES

9/12/2014 76

References & Related Links • http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc • https://www.docker.io/ • http://sysbench.sourceforge.net/ • http://dag.wiee.rs/home-made/dstat/ • http://www.openstack.org/ • https://wiki.openstack.org/wiki/Rally • https://wiki.openstack.org/wiki/Docker • http://devstack.org/ • http://www.linux-kvm.org/page/Main_Page • https://github.com/stackforge/nova-docker • https://github.com/dotcloud/docker-registry • http://www.netperf.org/netperf/ • http://www.tokutek.com/products/iibench/ • http://www.brendangregg.com/activebenchmarking.html

9/12/2014 77

REFERENCE

9/12/2014 78

Rally Boot 15 VM Configuration

9/12/2014 79

• Rally config: { "NovaServers.boot_server": [ { "args": { "flavor_id": 2, "image_id": “IMAGE_ID" }, "runner": { "type": "constant", "times": 15, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ] }

• Flavor +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True | | 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True | | 42 | m1.nano | 64 | 0 | 0 | | 1 | 1.0 | True | | 451 | m1.heat | 1024 | 0 | 0 | | 2 | 1.0 | True | | 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | | 6 | m1.perf | 4096 | 200 | 0 | | 2 | 1.0 | True | | 7 | m1.sql | 512 | 5 | 0 | | 1 | 1.0 | True | | 8 | m1.bench | 512 | 10 | 0 | | 1 | 1.0 | True | | 84 | m1.micro | 128 | 0 | 0 | | 1 | 1.0 | True | | 9 | m1.mini | 512 | 20 | 0 | | 1 | 1.0 | True | +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

Cloudy Benchmark: Serially Boot 15 VMs

• Docker +------------------+-------+--------------+-------------+---------------+---------------+---------------+

| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |

+------------------+-------+--------------+-------------+---------------+---------------+---------------+

| nova.boot_server | 15 | 4.8055100441 | 3.900927941 | 3.64957404137 | 4.56917948723 | 4.80114896297 |

+------------------+-------+--------------+-------------+---------------+---------------+---------------+

+---------------+---------------+--------------+---------------+---------------+---------------+-------------+

| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |

+---------------+---------------+--------------+---------------+---------------+---------------+-------------+

| 5.02777004242 | 4.11018741926 | 3.8767888546 | 4.77419886589 | 5.02389762402 | 1.0 | 15 |

+---------------+---------------+--------------+---------------+---------------+---------------+-------------+

• KVM +------------------+-------+---------------+---------------+---------------+---------------+---------------+

| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |

+------------------+-------+---------------+---------------+---------------+---------------+---------------+

| nova.boot_server | 15 | 7.00468301773 | 5.88419742584 | 4.84723997116 | 6.13021831512 | 6.48467411995 |

+------------------+-------+---------------+---------------+---------------+---------------+---------------+

+---------------+--------------+---------------+--------------+---------------+---------------+-------------+

| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |

+---------------+--------------+---------------+--------------+---------------+---------------+-------------+

| 7.22686600685 | 6.1018067201 | 5.05242800713 | 6.332523489 | 6.68914074898 | 1.0 | 15 |

+---------------+--------------+---------------+--------------+---------------+---------------+-------------+

9/12/2014 80

Rally Reboot 5x5 Configuration

9/12/2014 81

• Rally config: {

"NovaServers.boot_and_bounce_server": [

{

"args": {

"flavor_id": 2,

"image_id": “ID",

"actions": [

{"soft_reboot": 5}

]

},

"runner": {

"type": "constant",

"times": 5,

"active_users": 1

},

"context": {

"users": {

"tenants": 1,

"users_per_tenant": 1

}

}

}

]

}

Cloudy Performance: Serial VM Reboot • Docker +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.reboot_server | 25 | 6.85025811195 | 6.59131344795 | 6.50809788704 | 6.65774655342 | 6.72213015556 | | nova.boot_server | 5 | 4.96266412735 | 4.13350987434 | 3.82891011238 | 4.61819009781 | 4.79042711258 | | nova.delete_server | 5 | 8.74263191223 | 7.85760216713 | 6.61050701141 | 8.72944793701 | 8.73603992462 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | 45.8827331066 | 44.9664116383 | 43.5658369064 | 45.7298994541 | 45.8063162804 | 1.0 | 5 | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+

• KVM +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.reboot_server | 10 | 124.951740026 | 124.452507925 | 123.876129866 | 124.950992012 | 124.951366019 | | nova.boot_server | 2 | 6.1228749752 | 5.58395600319 | 5.04503703117 | 6.0150911808 | 6.068983078 | | nova.delete_server | 2 | 4.45799517632 | 4.45187151432 | 4.44574785233 | 4.45677044392 | 4.45738281012 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ | 632.718183994 | 632.315432072 | 631.912680149 | 632.63763361 | 632.677908802 | 0.4 | 5 | +---------------+---------------+---------------+--------------+---------------+---------------+-------------+

9/12/2014 82

Rally Snapshot Configuration

9/12/2014 83

Rally config: {

"NovaServers.boot_and_bounce_server": [

{

"args": {

"flavor_id": 1,

"image_id": “IMAGE",

"actions": [

{"soft_reboot": 5}

]

},

"runner": {

"type": "constant",

"times": 5,

"active_users": 1

},

"context": {

"users": {

"tenants": 1,

"users_per_tenant": 1

}

}

}

]

}

Cloudy Performance: Snapshot VM To Image

• Docker (note -- defect deleting image) +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.create_image | 1 | 26.3947799206 | 26.3947799206 | 26.3947799206 | 26.3947799206 | 26.3947799206 | | nova.boot_server | 2 | 4.120429039 | 3.85578501225 | 3.59114098549 | 4.06750023365 | 4.09396463633 | | nova.delete_server | 2 | 8.61395692825 | 7.64416801929 | 6.67437911034 | 8.41999914646 | 8.51697803736 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ | n/a | n/a | n/a | n/a | n/a | 0 | 1 | +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ • KVM +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ | nova.delete_image | 1 | 0.700344085693 | 0.700344085693 | 0.700344085693 | 0.700344085693 | 0.700344085693 | | nova.create_image | 1 | 42.92771101 | 42.92771101 | 42.92771101 | 42.92771101 | 42.92771101 | | nova.boot_server | 2 | 40.9650099277 | 22.9950100183 | 5.02501010895 | 37.3710099459 | 39.1680099368 | | nova.delete_server | 2 | 4.47270512581 | 4.46178817749 | 4.45087122917 | 4.47052173615 | 4.47161343098 | +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ | 98.541975975 | 98.541975975 | 98.541975975 | 98.541975975 | 98.541975975 | 1.0 | 1 | +--------------+--------------+--------------+--------------+---------------+---------------+-------------+

9/12/2014 84

Configuring Docker LXC Container For 2CPU x 4G RAM

• Pin container to 2 CPUs / Mems – Create cpuset cgroup – Pin group to cpuset.mems to 0,1 – Pin group to cpuset.cpus to 0,1 – Add container root proc to tasks

• Limit container memory to 4G – Create memory cgroup – Set memory.limit_in_bytes to 4G – Add container root proc to tasks

9/12/2014 85

Docker Command Line

• Start docker image from docker CLI root@devstack-compute:~/devstack$ time docker run -d guillermo/mysql

859f2a88adeb7190eb855eb060172fc2f137bff609f2f3be8ad8ee069d7e88f4

real 0m0.191s

user 0m0.004s

sys 0m0.004s

9/12/2014 86