linux-based virtualization for hpc clusters · linux-based virtualization for hpc clusters lucas...
TRANSCRIPT
Linux-based virtualization for HPC clusters
Lucas Nussbaum, Fabienne Anhalt,Olivier Mornard, Jean-Patrick Gelas
Laboratoire de l’Informatique et du ParallélismeINRIA - UCB Lyon 1 - ENS Lyon - CNRS
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 1 / 27
Introduction
Virtualization :
Subject of a lot of attention in the recent years
Solves many problems in classic datacenters
But limited adoption in High Performance Computing
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 2 / 27
Virtualization in HPC
Pros :
Dynamic allocation of resources to jobe.g automatically scale down a job from 100 to 10 machines
Easier to share resources between different jobsMix I/O-intensive with CPU-intensive jobs⇒ maximize usage
Easy checkpointing of jobsSolves a long-term problem in HPC
Deployment of job-specific work environmentUser no longer limited by administrator’s choices
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 3 / 27
Virtualization in HPC
Cons :
Overhead caused by the virtualization layerPoorly understood, rarely evaluated
Non-exclusive access to hardware
Performance may vary
Harder to make use of HPC-specific devicesHigh Performance networks (Infiniband, Myrinet)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 4 / 27
Context of this work : HIPCAL Project
Research project funded by ANR (= French NSF)
Build virtualized infrastructures= virtual clusters spread over the Grid
Combine system virtualization and network virtualizationmultiple virtual nodes per physical node
multiple virtual overlay networks on a shared longdistance communication infrastructure
Opportunity to allocate bandwidth per user(performance guarantees)
Software : HIPerNET (still in development)
http ://hipcal.lri.fr/
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 5 / 27
Goal of this work
Compare and evaluate two virtualization solutions
Xen
KVM
In the context of High Performance Computing
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 6 / 27
Xen
Virtual Machine hypervisorFirst public release : 2003Usually uses Linux as dom0Developed outside Linux by XenSource (now Citrix)
DomU support in vanilla kernelDom0 merge under discussionDistributions forward-port the Xen patch
Several distributions have stopped supporting Xen recently,and focused on KVM
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 7 / 27
KVM
Virtualization infrastructure integrated in Linux= Linux as an hypervisor
Requires hardware virtualization support in CPU
First release in 2007 (2.6.20)
Developed by Qumranet (acquired by Red Hat)
Relies on QEMU for I/OOriginally only QEMU emulated devicesNow paravirtualization with virtio (net, disk)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 8 / 27
KVM : I/O path
Apps
Hardware with virtualization acceleration
KVM module
Qemu I/OLinuxguest Linux
(1)
(2)(3) Linux kernel
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 9 / 27
Evaluation : Benchmarks
Micro-benchmarks :
CPUDiskNetwork
HPC Challenge benchmarks :Covers various aspects of High Performance ComputingUsed for the HPC Challenge @ SuperComputingIncludes Linpack/HPL (used by Top500)Other benchmarks included : PTRANS, STREAM, LBB, ...
http://icl.cs.utk.edu/hpcc/
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 10 / 27
Evaluation : Setup
32-nodes clusterDell PowerEdge 1950 (2 Dual-core Xen 5148 LV ; 8 GB RAM)
Same userspace, based on Debian sidDifferent kernels (host and guest)4 configurations evaluated :
Xen with full (hardware) virtualization (Xen FV)Xen with paravirtualization (Xen PV)Standard KVM, using Qemu devices (KVM FV)KVM with virtio-based paravirtualization (KVM PV)
Software versions :Xen 3.3.1 + Linux 2.6.18 (from XenSource)Linux 2.6.29 + KVM 84
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 11 / 27
CPU
CPU-intensive synthetic benchmark
Using 4 VCPU (mapped on 4 real CPU)
Almost no memory used
Compared with time on host system (2.6.29)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 12 / 27
CPU
100
100.5
101
101.5
102
Host (Xen) KVM Xen FV Xen PV
Per
form
ance
diff
. with
KV
M H
ost (
%)
Similar results : almost no overhead
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 12 / 27
Disk
Large file written using dd
Varying block sizes
Confirmed with bonnie++
Host system using RAID-0 (expected 120 MB/s)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 13 / 27
Disk
0
50
100
150
200
64 512 4k 32k 256k
Dis
k w
rite
spee
d (M
B/s
)
Write block size (B)
Host (2.6.29)Xen FVXen PV
KVM FVKVM PV
KVM-85 PV
Xen acks writes before they are committed to disk ?Bad KVM+virtio performance
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 13 / 27
Network
Throughput measured using iperf
KVM FV : e1000 NIC emulation
Xen FV ; Realtek 8139 (no GbE NIC available)
3 configurations evaluated :Communication between 2 VMs on the same machineCommunication between a VM and a remote systemScalability (several VMs, several remote systems)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 14 / 27
Network : inter-VM communications
FV PVKVM 648.3 813.2Xen 96.05 4451
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 15 / 27
Network : sending/receiving to remote hostreceivingsending
VMVM
physical machinephysical machine
0
100
200
300
400
500
600
700
800
900
1000
KVM FV Xen FV KVM PV Xen PV
Thr
ough
put (
Mbi
t/s)
SendingReceiving
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 16 / 27
Network : several VM
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 17 / 27
Network : several VM - sending
0 100 200 300 400 500 600 700 800 900
1000 1100 1200 1300
KVM FV Xen FV KVM PV Xen PV
Thr
ough
put (
Mbi
t/s)
1 VM2 VMs4 VMs8 VMs
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 18 / 27
Network : several VM - receiving
0 100 200 300 400 500 600 700 800 900
1000 1100 1200 1300
KVM FV Xen FV KVM PV Xen PV
Thr
ough
put (
Mbi
t/s)
1 VM2 VMs4 VMs8 VMs
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 19 / 27
Micro-benchmarks - summary
CPU : similar resultsDisk : Xen better (but cheats ?), KVM+virtio has problemsNetwork :
Xen PV very goodKVM+virtio OKstandard KVM a bit slowerXen FV extremely slow
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 20 / 27
HPCC benchmarks
7 benchmarks used by the HPC Challenge3 used in this presentation
Goal : evaluate all aspects of clusters
8 different configurations :32 host system - 2.6.29 + KVM32 dom0 - 2.6.1832 KVM VM with 4 CPU each, using virtio128 KVM VM with 1 CPU each, using virtio32 paravirtualized Xen VM, 4 CPU each128 paravirtualized Xen VM, 1 CPU each32 Xen VM with full virtualization, 4 CPU each128 Xen VM with full virtualization, 1 CPU each
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 21 / 27
HPCC benchmarks
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 22 / 27
PTRANS benchmark
Sends large messages between nodes
Measures the total communication capacity
⇒ Result : aggregate bandwidth
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 23 / 27
PTRANS benchmark
0
0.5
1
1.5
2
2.5
3
Hos
t (K
VM
)
Hos
t (X
en)
KV
M (
1V/4
C)
KV
M (
4V/1
C)
Xen
PV
(1V
/4C
)
Xen
PV
(4V
/1C
)
Xen
FV
(1V
/4C
)
Xen
FV
(4V
/1C
)Agg
rega
te b
andw
idth
(G
B/s
)
4 KVM VM per node⇒ better spread the I/O load ?Poor Xen performance
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 23 / 27
Latency and Bandwidth benchmark
Messages are being sent in a ring (ping-pong)
Two sizes : 8 B and 2-MB
Results : average node-to-node latency and bandwidth
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 24 / 27
Latency and Bandwidth benchmark
0
50
100
150
200
250
300
Hos
t (K
VM
)
Hos
t (X
en)
KV
M (
1V/4
C)
KV
M (
4V/1
C)
Xen
PV
(1V
/4C
)
Xen
PV
(4V
/1C
)
Late
ncy
(us)
Latency : higher overhead with KVM
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 24 / 27
Latency and Bandwidth benchmark
0
5
10
15
20
25
Hos
t (K
VM
)
Hos
t (X
en)
KV
M (
1V/4
C)
KV
M (
4V/1
C)
Xen
PV
(1V
/4C
)
Xen
PV
(4V
/1C
)
Xen
FV
(1V
/4C
)
Xen
FV
(4V
/1C
)
Ban
dwid
th (
MB
/s)
Bandwidth : better bandwidth with KVM (4 VM / node)
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 24 / 27
Linpack/HPL
Combines computation and communicationGives a global picture
Used for Top500
Sensitive to latency
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 25 / 27
Linpack/HPL
0
50
100
150
200
250
Hos
t (K
VM
)
Hos
t (X
en)
KV
M (
1V/4
C)
KV
M (
4V/1
C)
Xen
PV
(1V
/4C
)
Xen
PV
(4V
/1C
)
Xen
FV
(1V
/4C
)
Xen
FV
(4V
/1C
)
Per
form
ance
(G
Flo
p/s)
Host : 223 Gflop/s
KVM 1V/N : 154 (-31%)
KVM 4V/N : 152 (-32%)
Xen 1V/N : 235 (+5%)
Xen 4V/N : 183 (-18%)
Xen outperforms all other solutionsEven faster than host with 1 VM / node
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 25 / 27
Conclusion
KVM and Xen : different performance characteristicsXen+paravirtualization generally faster
KVM : still has some rough edges
Xen : poor support from distribution, harder to setup
Future work :
Other aspects of virtualization : fairness,checkpoint/migration
PCI pass-through for high performance networks
Re-evaluate with new releases : Xen 3.4, KVM-88
Linux Containers : promising alternative ?
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 26 / 27
Linux-based virtualization for HPC clusters
Lucas Nussbaum, Fabienne Anhalt,Olivier Mornard, Jean-Patrick Gelas
Laboratoire de l’Informatique et du ParallélismeINRIA - UCB Lyon 1 - ENS Lyon - CNRS
Lucas Nussbaum, Fabienne Anhalt, Olivier Mornard, J-P Gelas Linux-based virtualization for HPC clusters 27 / 27