kvm and docker lxc benchmarking with openstack

Passive Benchmarking with docker LXC, KVM & OpenStackHosted @ SoftLayer

Boden Russell ([email protected])IBM Global Technology Services

Advanced Cloud Solutions & Innovation

04/07/2023 2

Why Linux Containers (LXC) Fast– Runtime performance near bare metal speeds– Management operations (run, stop , start, etc.) in seconds / milliseconds

Agile– VM-like agility – it’s still “virtualization”– Seamlessly “migrate” between virtual and bare metal environments

Flexible– Containerize a “system”– Containerize “application(s)”

Lightweight– Just enough Operating System (JeOS)– Minimal per container penalty

Inexpensive– Open source – free – lower TCO– Supported with out-of-the-box modern Linux kernel

Ecosystem– Growing in popularity– Vibrant community & numerous 3rd party apps

04/07/2023 3

Hypervisors vs. Linux Containers

Hardware

Operating System

Hypervisor

Virtual Machine

Operating System

Bins / libs

App App

Virtual Machine

Operating System

Bins / libs

App App

Hardware

Hypervisor

Virtual Machine

Operating System

Bins / libs

App App

Virtual Machine

Operating System

Bins / libs

App App

Hardware

Operating System

Container

Bins / libs

App App

Container

Bins / libs

App App

Type 1 Hypervisor Type 2 Hypervisor Linux Containers

Containers share the OS kernel of the host and thus are lightweight.However, each container must have the same OS kernel.

Containers are isolated, but share OS and, where appropriate, libs / bins.

04/07/2023 4

Hypervisor VM vs. LXC vs. Docker LXC

04/07/2023 5

Docker in OpenStack Havana– Nova virt driver which integrates with docker REST API on backend– Glance translator to integrate docker images with Glance

Icehouse– Heat plugin for docker

Both options are still under development

nova-docker virt driver docker heat plugin

04/07/2023 6

About This Benchmark Use case perspective– As an OpenStack Cloud user I want a Ubuntu based VM with MySQL… Why would I choose

docker LXC vs a traditional hypervisor? OpenStack “Cloudy” perspective– LXC vs. traditional VM from a Cloudy (OpenStack) perspective– VM operational times (boot, start, stop, snapshot)– Compute node resource usage (per VM penalty); density factor

Guest runtime perspective– CPU, memory, file I/O, MySQL OLTP

Why KVM?– Exceptional performance

DISCLAIMERSThe tests herein are “passive” – no in depth tuning, analysis, etc. More active testing is warranted. These results do not necessary reflect your

workload or exact performance nor are they guaranteed to be statistically sound.

04/07/2023 7

Benchmark Environment Topology @ SoftLayer

glance api / reg

nova api / cond / etc

keystone

…

rally


cinder api / sch / vol

docker lxc

dstat

controller compute node

glance api / reg


keystone

…

rally


cinder api / sch / vol

KVM

dstat

controller compute node

+Awesome!

+Awesome!

04/07/2023 8

Benchmark SpecsSpec Controller Node (4CPU x 8G RAM) Compute Node (16CPU x 96G RAM)

Environment Bare Metal @ SoftLayer Bare Metal @ SoftLayer

Mother Board SuperMicro X8SIE-F Intel Xeon QuadCore SingleProc SATA [1Proc]

SuperMicro X8DTU-F_R2 Intel Xeon HexCore DualProc [2Proc]

CPU Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] (Intel Xeon-Westmere 5620-Quadcore [2.4GHz]) x 2

Memory (Kingston 4GB DDR3 2Rx8 4GB DDR3 2Rx8 [4GB]) x2 (Kingston 16GB DDR3 2Rx4 16GB DDR3 2Rx4 [16GB]) x 6

HDD (LOCAL) Digital WD Caviar RE3 WD5002ABYS [500GB]; SATAII Western Digital WD Caviar RE4 WD5003ABYX [500GB]; SATAII

NIC eth0/eth1 @ 100 Mbps eth0/eth1 @100 Mbps

Operating System Ubuntu 12.04 LTS 64bit Ubuntu 12.04 LTS 64bit

Kernel 3.5.0-48-generic 3.8.0-38-generic

IO Scheduler deadline deadline

Hypervisor tested NA - KVM 1.0 + virtio + KSM (memory deduplication)- docker 0.10.0 + go1.2.1 + commit dc9c28f + AUFS

OpenStack Trunk master via devstack Trunk master via devstack. Libvirt KVM nova driver / nova-docker virt driver

OpenStack Benchmark Client

OpenStack project rally NA

Metrics Collection NA dstat

Guest Benchmark Driver NA - Sysbench 0.4.12- mbw 1.1.1.-2- iibench (py)- netperf 2.5.0-1

VM Image NA - Scenario 1 (KVM): official ubuntu 12.04 image + mysql snapshotted and exported to qcow2 – 1080 MB

- Scenario 2 (docker): guillermo/mysql -- 334.8 MB

Hosted @

https://www.docker.io/

https://wiki.openstack.org/wiki/Rally


http://dag.wiee.rs/home-made/dstat/

http://sysbench.sourceforge.net/

http://manpages.ubuntu.com/manpages/lucid/man1/mbw.1.html

http://www.tokutek.com/products/iibench/

http://www.netperf.org/netperf/

http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

https://index.docker.io/u/guillermo/mysql/

https://index.docker.io/u/guillermo/mysql/

04/07/2023 9

Test Descriptions: Cloudy BenchmarksBenchmark Benchmark Driver Description

OpenStack Cloudy Benchmarks

Serial VM boot(15 VMs)

OpenStack Rally - Boot VM from image- Wait for ACTIVE state- Repeat the above a total of 15 times- Delete VMs

VM reboot(5 VMs rebooted 5 times each)

OpenStack Rally - Boot VM from image- Wait for ACTIVE state- Soft reboot VM 5 times- Delete VM- Repeat the above a total of 5 times

VM snapshot(1 VM, 1 snapshot)

OpenStack Rally - Boot VM from image- Wait for ACTIVE state- Snapshot VM to glance image- Delete VM

04/07/2023 10

Test Descriptions: Guest BenchmarksBenchmark Benchmark Driver Description

Guest Runtime Benchmarks

CPU performance Sysbench from within the guest - Run sysbench cpu test- Repeat a total of 10 times- Average results over the 10 times

OLTP (MySQL) performance

Sysbench from within the guest - Run sysbench OLTP test- Repeat a total of 10 times- Average results over the 10 times

MySQL Indexed insertion benchmark - Run iibench for a total of 1M inserts printing stats at 100K intervals

- Collect data over 5 runs & average

File I/O performance Sysbench from within the guest- Synchronous IO- No-direct

- Run sysbench OLTP test- Repeat a total of 10 times- Average results over the 10 times

Memory performance

Mbw from within the guest - Run mbw with array size of 1000 MiB and each test 10 times

- Collect average over 10 runs per test

Network performance

Netperf - Run netperf server on controller- From guest run netperf client in IPv4 mode- Repeat text 5x- Average results

04/07/2023 11

SERIALLY BOOT 15 VMS

OpenStack Cloudy Benchmark

04/07/2023 12

Cloudy Performance: Serial VM Boot Benchmark scenario overview– Boot VM via OpenStack nova– Wait for VM to become active– Repeat the above steps for a total of 15 VMs– Delete all VMs

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

2

4

6

8

10

12

14

16

Benchmark Visualization

VMs

Time

Activ

e VM

s

04/07/2023 13

Cloudy Performance: Serial VM Boot

docker KVM0

1

2

3

4

5

6

7

3.900927941

5.88419742584

Average Server Boot Time

Series1

Tim

e in

Sec

onds

Docker 1.5x faster

04/07/2023 14


1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 880

102030405060708090

100

Docker: Compute Node CPU

usrsys

Time

CPU

Usag

e In

Per

cent

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 1360

102030405060708090

100

KVM: Compute Node CPU

usrsys

Time

CPU

Usag

e In

Per

cent

Averages

– 1.14

– 0.44

Averages

– 12.6

– 2.08

04/07/2023 15


1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 880.00E+005.00E+081.00E+091.50E+092.00E+092.50E+093.00E+093.50E+094.00E+094.50E+095.00E+09

Docker: Compute Node Used Memory

Memory

Time

Mem

ory

Used

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 1360.00E+005.00E+081.00E+091.50E+092.00E+092.50E+093.00E+093.50E+094.00E+094.50E+095.00E+09

KVM: Compute Node Used Memory

Memory

Time

Mem

ory

Used

Delta687 MB

Per VM45.8 MB

Delta2775 MB

Per VM185 MB

04/07/2023 16


1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 880

5

10

15

20

25

Docker: Compute Node 1m Load Average

1m

Time

1 M

inut

e Lo

ad A

vera

ge

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 1360

5

10

15

20

25

KVM: Compute Node 1m Load Average

1m

Time

1 M

inut

e Lo

ad A

vera

ge

Average

0.09 %

Average

9.94 %

04/07/2023 17

SERIAL VM SOFT REBOOT


04/07/2023 18

Cloudy Performance: Serial VM Reboot Benchmark scenario overview– Boot a VM• Wait for it to become active

– Soft reboot the VM• Wait for it to become active• Repeat soft reboot a total of 5 times

– Delete VM– Repeat the above for a total of 5 VMs

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 550

1

2

3

4

5

6

Benchmark Visualization

Active VMs

Time

Activ

e VM

s

04/07/2023 19

Cloudy Performance: Serial VM Reboot

docker KVM0

20

40

60

80

100

120

140

6.59131344795

124.452507925

Average Server Reboot Time

Series1

Tim

e In

Sec

onds

Docker 18.9x faster

04/07/2023 20


docker KVM0

1

2

3

4

5

6

7

8

9

7.85760216713

4.45187151432

Average Server Delete Time

Series1

Tim

e In

Sec

onds

KVM 1.75x faster

04/07/2023 21

Cloudy Performance: Serial VM Reboot After investigating docker delete times– Docker sends SIGTERM to container process to stop– Bash is immune to SIGTERM– Docker waits for X seconds before stop– Default container image (init) command was using bash– See: https://github.com/dotcloud/docker/issues/3766

Rebuild the docker mysql image– Don’t use bash for container image command– No change to docker (hypervisor); image config only change

Rerun rally reboot tests; Cloudy reboot round 2

https://github.com/dotcloud/docker/issues/3766

04/07/2023 22


docker KVM0

20

40

60

80

100

120

140

2.54194530487

124.452507925

Average Server Reboot Time (Round 2)

Series1

Tim

e In

Sec

onds

Docker 48.99x faster

04/07/2023 23


docker KVM0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

4.09325451851

4.45187151432

Average Server Delete Time (Round 2)

Series1

Tim

e In

Sec

onds

Docker 1.09x faster

04/07/2023 24


3 136 269 402 535 668 801 934 1067 1200 1333 1466 1599 1732 1865 1998 2131 2264 2397 2530 2663 2796 2929 30620123456789

10


usrsys

Time

CPU

Usag

e In

Per

cent

1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 2410123456789

10


usrsys

Time

CPU

Usag

e In

Per

cent

Averages

– 0.34

– 0.11

Averages

– 0.82

– 0.19

04/07/2023 25


1 9 17 25 33 41 49 57 65 73 81 89 97 1051131211291371451531611691771851932012092172252332410.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09


Memory

Time

Mem

ory

Used

3 143 283 423 563 703 843 983 1123126314031543168318231963210322432383252326632803294330830.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09


Memory

Time

Mem

ory

Used

Delta

57 MB

Delta

467 MB

04/07/2023 26


3 129 255 381 507 633 759 885 1011113712631389151516411767189320192145227123972523264927752901302731530

0.5

1

1.5

2

2.5

3

3.5


1m

Time

1 M

inut

e Lo

ad A

vera

ge

1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185 193 201 209 217 225 233 2410

0.5

1

1.5

2

2.5

3

3.5


1m

Time

1 M

inut

e Lo

ad A

vera

ge

Average

0.05 %

Average

0.35 %

04/07/2023 27

SNAPSHOT VM TO IMAGE


04/07/2023 28

Cloudy Performance: Snapshot VM To Image Benchmark scenario overview– Boot a VM– Wait for it to become active– Snapshot the VM– Wait for image to become active– Delete VM

04/07/2023 29

Cloudy Performance: Snapshot VM To Image

docker KVM0

5

10

15

20

25

30

35

40

45

50

26.3947799206

42.92771101

Average Snapshot Server Time

Series1

Tim

e (S

econ

ds)

Docker 1.62x faster

04/07/2023 30


1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 1130

1

2

3

4

5

6

7


usrsys

Time

CPU

Usag

e In

Per

cent

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 670

1

2

3

4

5

6

7


usrsys

Time

CPU

Usag

e In

Per

cent

Averages

– 0.4

– 0.11

Averages

– 1.58

– 1.07

04/07/2023 31


1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 1131650000000

1700000000

1750000000

1800000000

1850000000


Memory

Time

Mem

ory

Used

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 6718300000001840000000185000000018600000001870000000188000000018900000001900000000191000000019200000001930000000


Memory

Time

Mem

ory

Used

Delta

48 MB

Delta

114 MB

04/07/2023 32


1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 1130

0.1

0.2

0.3

0.4

0.5

0.6


1m

Time

1 M

inut

e Lo

ad A

vera

ge

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67-0.1

2.77555756156289E-17

0.1

0.2

0.3

0.4

0.5

0.6


1m

Time

1 M

inut

e Lo

ad A

vera

ge

Average

0.1 %

Average

0.37 %

04/07/2023 33

GUEST PERFORMANCE BENCHMARKS

Guest VM Benchmark

04/07/2023 34

Guest Performance: CPU Linux sysbench 0.4.12 cpu test Calculate prime numbers up to 20000 2 threads Instance size– 4G RAM– 2 CPU cores– 20G disk

04/07/2023 35

Guest Performance: CPU

Docker KVM Bare Metal0

2

4

6

8

10

12

14

1615.11 15.08 15.03

Calculate Primes

Series1

Seconds

04/07/2023 36

Guest Performance: Memory Linux mbw 1.1.1-2 Instance size– 2 CPU– 4G memory

Execution options– 10 runs; average– 1000 MiB

04/07/2023 37

Guest Performance: Memory

MEMCPY DUMB MCBLOCK0

2000

4000

6000

8000

10000

12000

14000

Memory Benchmark Performance

BareMetaldockerKVM

Memory Tests

MiB

/s

04/07/2023 38

Guest Performance: Network Netperf 2.5.0-1– Netserver running on controller– Netperf on guest– Run netperf 5 times & average results

Instance size– 2 CPU– 4G memory

Execution options– IPv4 / TCP

04/07/2023 39

Guest Performance: Network

docker KVM0

100

200

300

400

500

600

700

800

900

1000

Network Throughput

Series1

Thro

ughp

ut In

10^

6 bi

ts/s

econ

d

04/07/2023 40

Guest Performance: File I/O Linux sysbench 0.4.12 fileio test– Synchronous IO– Random read / write– Total file size of 150G– 16K block size

Thread variations: 1, 8, 16, 32 Instance size– 4G RAM– 2 CPU cores– 200G disk

KVM specs– Disk cache set to none– Virtio– Deadline scheduler (host & guest)

Docker specs– AUFS storage driver– Deadline scheduler

04/07/2023 41

Guest Performance: File I/O

1 8 16 320

50

100

150

200

250

300

File I/O: Read

DockerKVM

Threads

Mb

1 8 16 320

20406080

100120140160180200

File I/O: Write

DockerKVM

Threads

Mb

04/07/2023 42

Guest Performance: File I/O – Read / Write

1 8 16 320

200

400

600

800

1000

1200

1400

1600

File I/O: Transfer Rate

DockerKVM

Threads

Kb/s

ec

04/07/2023 43

Guest Performance: MySQL OLTP Linux sysbench 0.4.12 oltp test– Table size of 2,000,000– MySQL 5.5 (installed on Ubuntu 12.04 LTS with apt-get)

Variations– Number of threads– Read only & read / write

Instance size– 4G RAM– 2 CPU cores– 20G disk

04/07/2023 44

Guest Performance: MySQL OLTP (Read)

2 8 160

10000

20000

30000

40000

50000

60000

MySQL OLTP Read Transactions

DockerKVM

Threads

Tota

l Tra

nsac

tions

2 8 160

100000200000300000400000500000600000700000800000

MySQL OLTP Read Requests

DockerKVM

Threads

Requ

ests

04/07/2023 45

Guest Performance: MySQL OLTP (Read / Write)

2 16 32 640

5000

10000

15000

20000

25000

MySQL OLTP Read/Write Transactions

dockerKVM

Threads

Tota

l Tra

nsac

tions

2 16 32 640

50000

100000

150000

200000

250000

300000

350000

400000

MySQL OLTP Read/Write Requests

DockerKVM

Threads

Requ

ests

04/07/2023 46

Guest Performance: MySQL Indexed Insertion Indexed insertion benchmark (iibench python script)– A total of 1,000,000 insertions– Print stats at 100K intervals– Collect stats over 5 runs– Average

Instance size– 4G RAM– 2 CPU cores– 20G disk

04/07/2023 47

Guest Performance: MySQL Indexed Insertion

1 2 3 4 5 6 7 8 9 100

20

40

60

80

100

120

140

MySQL Indexed Insertion @ 100K Intervals

dockerkvm

Table Size In 100K Increments

Seco

nds P

er 1

00K

Inse

rtion

Bat

ch

04/07/2023 48

BENCHMARK OBSERVATIONS

04/07/2023 49

Docker LXC CPU Growth 26x Lower Than VM

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 700

5

10

15

20

25

30

35

f(x) = 0.00912569940189079 x + 0.734884194053208

f(x) = 0.237004371128586 x + 2.29925813820674

User CPU Growth Trend

dockerLinear (docker)KVMLinear (KVM)

CPU

Usag

e In

Per

cent

Slope– 0.0091

– 0.237

04/07/2023 50

Docker LXC Memory Growth 3x Lower Than VM

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 730.00E+00

5.00E+08

1.00E+09

1.50E+09

2.00E+09

2.50E+09

3.00E+09

3.50E+09

4.00E+09

4.50E+09

5.00E+09

f(x) = 11257691.2277667 x + 1243983088.22487

f(x) = 32597283.8749557 x + 1271949903.76939

Memory Usage Growth Trend

dockerLinear (docker)KVMLinear (KVM)

Mem

ory

Used

Slope– 1E+07

– 3E+07

04/07/2023 51

Observations Cloudy operations with Docker LXC outperform VM– 48x server reboot, 1.5x server boot, 1.62x server snapshot, etc.

Docker LXC density potential compared to VMs– 3x memory savings– 26x CPU savings– 3.22x smaller images in this test (note – image sizes can vary based on required packages)

Docker LXC containers run on bare metal– Greater / equivalent “in the VM” performance at the micro benchmark level

Micro benchmark results do not always reflect macro performance– Always benchmark your “real” workload

Docker image (init) command can impact performance– Bash ignores SIGTERM

Nova-docker virt driver and docker-registry components still under dev– Nice work, but room for improvement (python anyone?)

Real performance of Docker LXC ops capped my Cloud manager– Can start the SQL image from docker CLI in 0.191s

04/07/2023 52

REFERENCES

04/07/2023 53

References & Related Links http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc https://www.docker.io/ http://sysbench.sourceforge.net/ http://dag.wiee.rs/home-made/dstat/ http://www.openstack.org/ https://wiki.openstack.org/wiki/Rally https://wiki.openstack.org/wiki/Docker http://devstack.org/ http://www.linux-kvm.org/page/Main_Page https://github.com/stackforge/nova-docker https://github.com/dotcloud/docker-registry http://www.netperf.org/netperf/ http://www.tokutek.com/products/iibench/ http://www.brendangregg.com/activebenchmarking.html

http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc







http://www.openstack.org/

http://www.openstack.org/



https://wiki.openstack.org/wiki/Docker

https://wiki.openstack.org/wiki/Docker

http://devstack.org/

http://devstack.org/

http://www.linux-kvm.org/page/Main_Page

http://www.linux-kvm.org/page/Main_Page

https://github.com/stackforge/nova-docker

https://github.com/stackforge/nova-docker

https://github.com/dotcloud/docker-registry

https://github.com/dotcloud/docker-registry





http://www.brendangregg.com/activebenchmarking.html

http://www.brendangregg.com/activebenchmarking.html

04/07/2023 54

REFERENCE

04/07/2023 55

Rally Boot 15 VM Configuration Rally config:{ "NovaServers.boot_server": [ { "args": { "flavor_id": 2, "image_id": “IMAGE_ID" }, "runner": { "type": "constant", "times": 15, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ]}

Flavor+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True || 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True || 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True || 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True || 42 | m1.nano | 64 | 0 | 0 | | 1 | 1.0 | True || 451 | m1.heat | 1024 | 0 | 0 | | 2 | 1.0 | True || 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True || 6 | m1.perf | 4096 | 200 | 0 | | 2 | 1.0 | True || 7 | m1.sql | 512 | 5 | 0 | | 1 | 1.0 | True || 8 | m1.bench | 512 | 10 | 0 | | 1 | 1.0 | True || 84 | m1.micro | 128 | 0 | 0 | | 1 | 1.0 | True || 9 | m1.mini | 512 | 20 | 0 | | 1 | 1.0 | True |+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

04/07/2023 56

Cloudy Benchmark: Serially Boot 15 VMs Docker+------------------+-------+--------------+-------------+---------------+---------------+---------------+| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |+------------------+-------+--------------+-------------+---------------+---------------+---------------+| nova.boot_server | 15 | 4.8055100441 | 3.900927941 | 3.64957404137 | 4.56917948723 | 4.80114896297 |+------------------+-------+--------------+-------------+---------------+---------------+---------------+

+---------------+---------------+--------------+---------------+---------------+---------------+-------------+| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |+---------------+---------------+--------------+---------------+---------------+---------------+-------------+| 5.02777004242 | 4.11018741926 | 3.8767888546 | 4.77419886589 | 5.02389762402 | 1.0 | 15 |+---------------+---------------+--------------+---------------+---------------+---------------+-------------+

KVM+------------------+-------+---------------+---------------+---------------+---------------+---------------+| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |+------------------+-------+---------------+---------------+---------------+---------------+---------------+| nova.boot_server | 15 | 7.00468301773 | 5.88419742584 | 4.84723997116 | 6.13021831512 | 6.48467411995 |+------------------+-------+---------------+---------------+---------------+---------------+---------------+

+---------------+--------------+---------------+--------------+---------------+---------------+-------------+| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |+---------------+--------------+---------------+--------------+---------------+---------------+-------------+| 7.22686600685 | 6.1018067201 | 5.05242800713 | 6.332523489 | 6.68914074898 | 1.0 | 15 |+---------------+--------------+---------------+--------------+---------------+---------------+-------------+

04/07/2023 57

Rally Reboot 5x5 Configuration Rally config:{ "NovaServers.boot_and_bounce_server": [ { "args": { "flavor_id": 2, "image_id": “ID", "actions": [ {"soft_reboot": 5} ] }, "runner": { "type": "constant", "times": 5, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ]}

04/07/2023 58

Cloudy Performance: Serial VM Reboot Docker+--------------------+-------+---------------+---------------+---------------+---------------+---------------+| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |+--------------------+-------+---------------+---------------+---------------+---------------+---------------+| nova.reboot_server | 25 | 6.85025811195 | 6.59131344795 | 6.50809788704 | 6.65774655342 | 6.72213015556 || nova.boot_server | 5 | 4.96266412735 | 4.13350987434 | 3.82891011238 | 4.61819009781 | 4.79042711258 || nova.delete_server | 5 | 8.74263191223 | 7.85760216713 | 6.61050701141 | 8.72944793701 | 8.73603992462 |+--------------------+-------+---------------+---------------+---------------+---------------+---------------+

+---------------+---------------+---------------+---------------+---------------+---------------+-------------+| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |+---------------+---------------+---------------+---------------+---------------+---------------+-------------+| 45.8827331066 | 44.9664116383 | 43.5658369064 | 45.7298994541 | 45.8063162804 | 1.0 | 5 |+---------------+---------------+---------------+---------------+---------------+---------------+-------------+

KVM+--------------------+-------+---------------+---------------+---------------+---------------+---------------+| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |+--------------------+-------+---------------+---------------+---------------+---------------+---------------+| nova.reboot_server | 10 | 124.951740026 | 124.452507925 | 123.876129866 | 124.950992012 | 124.951366019 || nova.boot_server | 2 | 6.1228749752 | 5.58395600319 | 5.04503703117 | 6.0150911808 | 6.068983078 || nova.delete_server | 2 | 4.45799517632 | 4.45187151432 | 4.44574785233 | 4.45677044392 | 4.45738281012 |+--------------------+-------+---------------+---------------+---------------+---------------+---------------+

+---------------+---------------+---------------+--------------+---------------+---------------+-------------+| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |+---------------+---------------+---------------+--------------+---------------+---------------+-------------+| 632.718183994 | 632.315432072 | 631.912680149 | 632.63763361 | 632.677908802 | 0.4 | 5 |+---------------+---------------+---------------+--------------+---------------+---------------+-------------+

04/07/2023 59

Rally Snapshot Configuration

Rally config:{ "NovaServers.boot_and_bounce_server": [ { "args": { "flavor_id": 1, "image_id": “IMAGE", "actions": [ {"soft_reboot": 5} ] }, "runner": { "type": "constant", "times": 5, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ]}

04/07/2023 60

Cloudy Performance: Snapshot VM To Image Docker (note -- defect deleting image)+--------------------+-------+---------------+---------------+---------------+---------------+---------------+| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |+--------------------+-------+---------------+---------------+---------------+---------------+---------------+| nova.create_image | 1 | 26.3947799206 | 26.3947799206 | 26.3947799206 | 26.3947799206 | 26.3947799206 || nova.boot_server | 2 | 4.120429039 | 3.85578501225 | 3.59114098549 | 4.06750023365 | 4.09396463633 || nova.delete_server | 2 | 8.61395692825 | 7.64416801929 | 6.67437911034 | 8.41999914646 | 8.51697803736 |+--------------------+-------+---------------+---------------+---------------+---------------+---------------+

+-----------+-----------+-----------+--------------+---------------+---------------+-------------+| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |+-----------+-----------+-----------+--------------+---------------+---------------+-------------+| n/a | n/a | n/a | n/a | n/a | 0 | 1 |+-----------+-----------+-----------+--------------+---------------+---------------+-------------+

KVM+--------------------+-------+----------------+----------------+----------------+----------------+----------------+| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |+--------------------+-------+----------------+----------------+----------------+----------------+----------------+| nova.delete_image | 1 | 0.700344085693 | 0.700344085693 | 0.700344085693 | 0.700344085693 | 0.700344085693 || nova.create_image | 1 | 42.92771101 | 42.92771101 | 42.92771101 | 42.92771101 | 42.92771101 || nova.boot_server | 2 | 40.9650099277 | 22.9950100183 | 5.02501010895 | 37.3710099459 | 39.1680099368 || nova.delete_server | 2 | 4.47270512581 | 4.46178817749 | 4.45087122917 | 4.47052173615 | 4.47161343098 |+--------------------+-------+----------------+----------------+----------------+----------------+----------------+

+--------------+--------------+--------------+--------------+---------------+---------------+-------------+| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |+--------------+--------------+--------------+--------------+---------------+---------------+-------------+| 98.541975975 | 98.541975975 | 98.541975975 | 98.541975975 | 98.541975975 | 1.0 | 1 |+--------------+--------------+--------------+--------------+---------------+---------------+-------------+

04/07/2023 61

Configuring Docker LXC Container For 2CPU x 4G RAM

Pin container to 2 CPUs / Mems– Create cpuset cgroup– Pin group to cpuset.mems to 0,1– Pin group to cpuset.cpus to 0,1– Add container root proc to tasks

Limit container memory to 4G– Create memory cgroup– Set memory.limit_in_bytes to 4G– Add container root proc to tasks

04/07/2023 62

Docker Command Line Start docker image from docker CLI

root@devstack-compute:~/devstack$ time docker run -d guillermo/mysql859f2a88adeb7190eb855eb060172fc2f137bff609f2f3be8ad8ee069d7e88f4

real 0m0.191suser 0m0.004ssys 0m0.004s

kvm and docker lxc benchmarking with openstack

Technology

virtio deadline

100s thread

clear memory

compute node

full test

compute node

5112014 document

compute node