a comparison of performance between kvm and docker ... · containers share host kernel and ......

19
A comparison of performance between KVM and Docker instances in OpenStack High Energy Accelerator Research Organiza�on (KEK), Japan 1 HEPiX Fall 2015 Workshop at BNL Wataru Takase

Upload: others

Post on 12-Jun-2020

11 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

A  comparison  of  performance  between  KVM  and  Docker  instances  in  OpenStack

High  Energy  Accelerator  Research  Organiza�on  (KEK),  Japan  

1

HEPiX  Fall  2015  Workshop  at  BNL

Wataru  Takase  

Page 2: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

KEK  site  will  become  Cloudy

2

  Integrate  private  cloud  into  batch  service    Deploy  CVMFS  Stratum  0  and  1    Start  online  storage  service  

Batch  scheduler Mediator Private  cloud

Job

Resource  request Request  for  

launching  instances

Resource  pool

Launch  Instances

WN

WN

WN

WN

Job Batch  worker  nodes

WN

WN

WN

WN

CVMFS

mount mount

Resource  request

Talk  focuses  on  this  part

Page 3: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Performance  Inves�ga�on  of  Cloud

  Is  virtual  machine  performance  good?    What  about  container  technology?    What  about  concurrency  impact  on  performance?    Measured  KVM  and  Docker  performance  in  OpenStack  by using  Rally  – Cloud  performance  –  Instance  performance  

3

Page 4: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

KVM?  Docker?

4

OpenStack  (IaaS)

Docker  driver Libvirt  driver

h�ps://www.docker.com/wha�sdocker

Each  VM  runs  on  virtual  hardware Containers  share  host  kernel  and  hardware  

  KVM  :  VM  hypervisor    Docker  :  Container  manger

Bare  Metal  Driver

VMWare  Driver

Xen  API  Driver

Page 5: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

What  is  Rally?

  Benchmarking  tool  for  OpenStack  

  Generates  real  workload    Provides  more  than  100  test  scenarios:  –  Boot  server  and  migrate  it  –  Create  image  and  boot  server  

–  Create  volume  and  a�ach  to  server  

–  …  

5

--- NovaServers.boot_and_delete_server: - args: image: name: ”centos-cloud:7.1" flavor: name: "m1.xsmall" min_sleep: 60 max_sleep: 60 runner: type: "constant" times: 32 concurrency: 32 context: users: tenants: 1 users_per_tenant: 32

Example_of_rally_benchmark_input.yaml

Page 6: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Test  environment

6

Physical  server OS Kernel CPU CPU  cores

RAM  (GB)

Disk  (GB)

OpenStack  controller   CentOS  7.1.1503 3.10.0-­‐229 Intel(R)  Xeon(R)  CPU  

E5649  x2 24* 32 300

OpenStack  compute CentOS  7.1.1503 3.10.0-­‐229 Intel(R)  Xeon(R)  CPU  

E5-­‐2630  v3  x2 32* 64 3800

Rally CentOS  7.1.1503 3.10.0-­‐229 AMD  Opteron(TM)  Processor  6212 8 16 1700

OS Kernel vCPU RAM  (GB)

Disk  (GB)

CentOS  7.1.1503 3.10.0-­‐229 1 1.8 10

  OpenStack  Kilo  (RDO)    1  controller  +  1  compute  node    nova-­‐network    Rally  (2d874a7)    Sysbench  0.4.12  

Instance  image  and  flavor *  HT  is  enabled

Page 7: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Test  environment

7

File  system XFS  on  LVM  on  hardware  RAID  5

IO  scheduler Deadline

Clocksource TSC

QEMU 1.5.3

libvirt 1.2.8

Docker 1.6.2

Nova  Docker  driver nova-­‐docker  stable/kilo  (d556444)

Image  format qcow2

Block  device  deriver VirtIO

Cache  mode none

File  system XFS

Clocksource TSC

Storage  driver OverlayFS

Compute  node

KVM

Docker

Page 8: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Benchmark  Scenarios 1.  Measure  cloud  performance  

–  Boot  a  server  and  then  delete  2.  Measure  instance  performance  

–  Boot  a  server  and  run  Sysbench  (test=cpu)  –  Boot  a  server  and  run  Sysbench  (test=memory,  memory-­‐oper=read)  –  Boot  a  server  and  run  Sysbench  (test=memory,  memory-­‐oper=write)  –  Boot  a  server  and  run  Sysbench  (test=fileio,  file-­‐test-­‐mode=seqrd)  –  Boot  a  server  and  run  Sysbench  (test=fileio,  file-­‐test-­‐mode=rndrd)  –  Boot  a  server  and  run  Sysbench  (test=fileio,  file-­‐test-­‐mode=seqwr)  –  Boot  a  server  and  run  Sysbench  (test=fileio,  file-­‐test-­‐mode=rndwr)  

  Each  scenario  launches  32  instances  –  Change  number  of  concurrent  requests  from  1  to  32  

8

Page 9: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Boot  a  server  and  then  delete

9

Building OpenStack Request x  32

Building

OpenStack

Request

  Concurrent  requests:  32

Building

Building

Request

Request

...

...

Building Request

  Concurrent  requests:    1

Wai�ng  for  ping,  ssh

Networking  �me

Ac�ve

Wai�ng  for  ping,  ssh

Networking  �me

Wai�ng  for  ping,  ssh

Wai�ng  for  ping,  ssh

Wai�ng  for  ping,  ssh

Wait  60  sec Dele�ng

Dele�ng  �me Building  �me

Wait  60  sec

Wait  60  sec Wait  60  sec

Wait  60  sec

Dele�ng

Dele�ng

Dele�ng

Dele�ng

Dele�ng  �me Building  �me

Deleted

Page 10: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Build  a  server

10

  At  high  concurrency  KVM  is  around  20%  be�er  

N:  96  Error  bar:  SD

Page 11: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

11

Wait  for  Networking

Boot  OS  +  cloud-­‐init

N:  96  Error  bar:  SD

cloud-­‐init  + Start  sshd

Page 12: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Building  +  Networking

12

N:  96  Error  bar:  SD

Page 13: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

13

Delete  a  server N:  96  Error  bar:  SD

Page 14: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Instance  Performance  Comparison

14

Wai�ng  for  ping,  ssh Building OpenStack Request

x  32

Wai�ng  for  ping,  ssh Building

OpenStack

Request

  Concurrent  requests:  32

Wai�ng  for  ping,  ssh Building

Wai�ng  for  ping,  ssh Building

Request

Request

...

...

Wai�ng  for  ping,  ssh Building

Request

  Concurrent  requests:  1

Sysbench

Sysbench

Sysbench

Sysbench

Sysbench

Wait

Measure  performance

Dele�ng

Wait

Wait

Wait

Wait

Dele�ng

Dele�ng

Measure  performance

Page 15: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

test=cpu,  cpu-­‐max-­‐prime=20000,  num-­‐threads=1

15

  At  low  concurrency  KVM  is  2-­‐7%  worse  than  na�ve    If  No.  of  concurrent  requests  >  2,  Docker  is  2%  worse  than  na�ve  

N:  32  Error  bar:  SD

Page 16: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

16

  test=memory,  memory-­‐oper=read,  memory-­‐block-­‐size=1K,  memory-­‐total-­‐size=100G,  max-­‐�me=300,  num-­‐threads=1  

  test=memory,  memory-­‐oper=write,  memory-­‐block-­‐size=1K,  memory-­‐total-­‐size=100G,  max-­‐�me=300,  num-­‐threads=1  

  At  low  concurrency  KVM  is  3-­‐10%  worse  than  na�ve    Docker  is  2-­‐5%  worse  than  na�ve  (concurrent  requests:  1-­‐16)  

N:  32  Error  bar:  SD

Page 17: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

17

  test=fileio,  file-­‐test-­‐mode=seqrd,  file-­‐block-­‐size=4K,  file-­‐total-­‐size=8G,  file-­‐num=128,  file-­‐extra-­‐flags=direct,  max-­‐�me=300,  num-­‐threads=1  

  test=fileio,  file-­‐test-­‐mode=rndrd,  file-­‐block-­‐size=4K,  file-­‐total-­‐size=8G,  file-­‐num=128,  file-­‐extra-­‐flags=direct,  max-­‐�me=300,  num-­‐threads=1  

  At  low  concurrency  KVM  sequen�al  read  is  60-­‐70%  worse  than  na�ve    KVM  random  read  is  several  %  worse  than  na�ve    Docker  achieves  na�ve  performance  

N:  32  Error  bar:  SD

Page 18: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

18

  test=fileio,  file-­‐test-­‐mode=seqwr,  file-­‐block-­‐size=4K,  file-­‐total-­‐size=8G,  file-­‐num=128,  file-­‐extra-­‐flags=direct,  max-­‐�me=300,  num-­‐threads=1  

  test=fileio,  file-­‐test-­‐mode=rndwr,  file-­‐block-­‐size=4K,  file-­‐total-­‐size=8G,  file-­‐num=128,  file-­‐extra-­‐flags=direct,  max-­‐�me=300,  num-­‐threads=1  

  At  low  concurrency  KVM  is  70-­‐80%  worse  than  na�ve    In  the  case  of  single  request,  Docker  sequen�al  write  is  15%  worse  than  na�ve    Beside  that  Docker  achieves  almost  na�ve  performance  

N:  32  Error  bar:  SD

Page 19: A comparison of performance between KVM and Docker ... · Containers share host kernel and ... OpenStack Building ping, ssh Request x 32 Wai ng for

Summary  and  Conclusion

19

  Cloud  performance  comparison  –  Docker  instance  becomes  ready  faster  than  KVM  (building  +  networking)  

  Instance  performance  comparison  –  CPU  and  memory  performance  

  Na�ve  >  Docker  >  KVM  –  KVM  

  File  IO  performance  is  poor  compared  to  na�ve  –  Docker  

  Read  performance  is  almost  the  same  as  na�ve    Write  performance  is  near  na�ve  

  Docker  seems  to  be  a  good  candidate  in  the  future  –  Nova-­‐docker  driver  lacks  some  features  and  has  some  bugs    

  More  inves�ga�on  is  needed  –  Security  –  Stability  –  Other  benchmarks  (network,  volume,  tuned  KVM)