qnibterminal plus infiniband - hpc-ai advisory council · • builds on-top linux containers ......
TRANSCRIPT
![Page 1: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/1.jpg)
QNIBTerminal plus InfiniBandContainerized MPI Workloads
2014-11-05Christian Kniep
![Page 2: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/2.jpg)
Agenda• About Me • Docker in a Nutshell • QNIBTerminal • Study
• Testbed + HPCG • Results
• Future Work • Conclusion
2
![Page 3: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/3.jpg)
• Hot topics • Containerization
• GO-Lang
• Log / Performance Management
• HPC Cluster Software Stack / Interconnect
• 1/2013 -> 10/2014 R&D @Bull SAS
• since then independent R&D / Freelancing
About Me
3
• Iteration through L1/2/3 SysOps
• Mostly german automotive sector
![Page 4: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/4.jpg)
Docker in a Nutshell
4
• (chroot on steroids)2
![Page 5: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/5.jpg)
• Builds on-top LinuX Containers (LXC)
• Kernel namespaces (isolation)
• cgroups (resource mgmt)
Docker in a Nutshell
5
• (chroot on steroids)2
![Page 6: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/6.jpg)
• intuitive build system
Docker in a Nutshell
6
• (chroot on steroids)2
• Builds on-top LinuX Containers (LXC)
• Kernel namespaces (isolation)
• cgroups (resource mgmt)
![Page 7: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/7.jpg)
• RedHat backing
• public repositories
• intuitive build system
Docker in a Nutshell
7
• (chroot on steroids)2
• Builds on-top LinuX Containers (LXC)
• Kernel namespaces (isolation)
• cgroups (resource mgmt)
![Page 8: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/8.jpg)
Traditional vs. Lightweight Layers
8
SERVER
HOST KERNEL
HYPERVISOR
KERNEL
SERVICE
Userland (OS)
KERNEL KERNEL
Userland (OS)Userland (OS) Userland (OS)
SERVICE SERVICE
SERVER
HOST KERNEL
SERVICE
Userland (OS)
Userland (OS)Userland (OS) Userland (OS)
SERVICE SERVICE
Traditional Virtualisation Containerisation
IB
IB
![Page 9: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/9.jpg)
Rocket ‚Docker‘• 2013-01-18 First commit • 2013-02-01 First online demo • 2013-03-21 Demo at PyCon US • 2013-03-23 Version 0.1• 2013-03-26 Repository on github.com created • 2013-04-23 Version 0.2• 2013-05-06 Version 0.3• 2013-06-03 Version 0.4• 2013-06-25 Joining Linux Foundation • 2013-07-18 Version 0.5 (top, mount) • 2013-08-23 Version 0.6 (-privileged, LXC conf) • 2013-09-09 Collaboration with Red Hat • 2013-10-29 Foundation of Docker Inc. • 2013-11-26 Version 0.7
9
• 2014-01-21 Docker Inc. raised $15M • 2014-02-04 Version 0.8 (MacOSX, btrfs exp., ONBUILD)• 2014-03-10 Version 0.9 (exec driver, libcontainer) • 2014-06-09 Version 1.0 (PROD, pause, XFS, COPY) • 2014-06-09 official repos, public repository • 2014-06-09 enterprise support, training, consulting • 2014-07-03 Version 1.1 (.dockerignore, mount „/“) • 2014-08-22 Version 1.2 (-restart, caps , rw /etc/hosts) • 2014-09-16 Docker Inc. raised $40M • 2014-09-16 Microsoft teams up with Docker Inc. • 2014-10-16 Version 1.3 (signed img, proc injection, …)
![Page 10: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/10.jpg)
QNIBTerminalMotivation
10
Plain Metrics
![Page 11: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/11.jpg)
QNIBTerminalMotivation
11
Plain Log Events
![Page 12: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/12.jpg)
QNIBTerminalMotivation
12
Overlap Metrics/Log Events
![Page 13: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/13.jpg)
QNIBTerminal Overview
13
haproxy haproxy
dnshelixdns
elk
kibana
logstash
etcd
carboncarbon
graphite-webgraphite-web
graphite-apigraphite-api
grafanagrafana
slurmctldslurmctld
compute0slurmd
compute<N>slurmd
Log/Events
Services Performance
Compute
elasticsearch
One Node Setup• All network traffic over bridge• Crippled MPI workload
![Page 14: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/14.jpg)
• Multiple Open MPI version installed
• gcc versions
• 3 containers on top (CentOS 6, CentOS 7, Ubuntu 12)
• SLURM Resource Scheduler
• 1 native partition
• 3 containers partitions
Testbed
14
• 8 nodes (CentOS 7, 2x 4core XEON, 32GB, Mellanox ConnectX-2)
![Page 15: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/15.jpg)
HPCG Benchmark• mimics thermodynamic application workload
• Linpack corrective / successor in the long-term?
15
![Page 16: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/16.jpg)
Resultspartition’s performance
16
GFL
OP/
s
3
3,75
4,5
5,25
6
native cos7 cos6 u12
CentOS 7.0 oMPI 1.6.4 gcc 4.8.2
![Page 17: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/17.jpg)
Resultspartition’s performance
17
GFL
OP/
s
3
3,75
4,5
5,25
6
native cos7 cos6 u12
CentOS 7.0 oMPI 1.6.4 gcc 4.8.2
CentOS 6.5 oMPI 1.5.4 gcc 4.4.7
Ubuntu12.04 oMPI 1.5.4 gcc 4.6.3
![Page 18: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/18.jpg)
Resultsmultiple MPI versions
18
GFL
OP/
s
3
3,75
4,5
5,25
6
distribution
nativecos7cos6u12
oMPI 1.6.4
oMPI 1.6.4
oMPI 1.5.4
oMPI 1.5.4
![Page 19: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/19.jpg)
Resultsmultiple MPI versions
19
GFL
OP/
s
3
3,75
4,5
5,25
6
distribution 1.6.4 1.8.4
nativecos7cos6u12
oMPI 1.6.4
oMPI 1.6.4
oMPI 1.5.4
oMPI 1.5.4
gcc 4.8.2gcc 4.8.2gcc 4.4.7gcc 4.6.3
![Page 20: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/20.jpg)
Resultsmultiple MPI versions
20
GFL
OP/
s
3
3,75
4,5
5,25
6
distribution 1.5.4 1.6.4 1.8.4
nativecos7cos6u12
oMPI 1.6.4
oMPI 1.6.4
oMPI 1.5.4
oMPI 1.5.4
gcc 4.8.2gcc 4.8.2gcc 4.4.7gcc 4.6.3
![Page 21: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/21.jpg)
• Security evaluations
• Compare different frameworks to orchestrate
• Use of SV-IOR (Keynote earlier today)
• Compare with tuned bare-metal
• Tune docker installation
Future Work
21
• Benchmark real-world applications
![Page 22: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/22.jpg)
• Out-of-the-box: container beats bare-metal
• Continuous testing/deployment of containerized workloads
• Bare-metal kernel provides access to IB
• Container in charge from MPI upwards
Conclusion
22
• Bunch of tooling within docker ecosystem
• Abstraction bare-metal / application works fine
• Low performance overhead
![Page 23: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/23.jpg)
• Contact • @CQnib / @qnibinc • [email protected] • http://qnib.org
La Fin
23
https://www.flickr.com/photos/dharmabum1964/3108162671
![Page 24: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/24.jpg)
• Paper: http://doc.qnib.org/
• Contact • @CQnib / @qnibinc • [email protected] • http://qnib.org
La Fin
24
https://www.flickr.com/photos/dharmabum1964/3108162671
![Page 25: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/25.jpg)
La Fin
25
https://www.flickr.com/photos/dharmabum1964/3108162671
• Interested? • Docker Pitch today • Internal Evaluations • Workshops / Talks
• Paper: http://doc.qnib.org/
• Contact • @CQnib / @qnibinc • [email protected] • http://qnib.org
![Page 26: QNIBTerminal plus InfiniBand - HPC-AI Advisory Council · • Builds on-top LinuX Containers ... • 2014-09-16 Microsoft teams up with Docker Inc. ... Plain Metrics. QNIBTerminal](https://reader033.vdocuments.site/reader033/viewer/2022042307/5ed2e8674e7ab45be80cee85/html5/thumbnails/26.jpg)
La Fin
26
https://www.flickr.com/photos/dharmabum1964/3108162671
• Interested? • Docker Pitch today • Internal Evaluations • Workshops / Talks
• Questions?
• Paper: http://doc.qnib.org/
• Contact • @CQnib / @qnibinc • [email protected] • http://qnib.org