Docker RPi2 challenge@damienduportal @ndeloof @YoannDubreuil @Quintus23M

Challenge Introduction

DockerCon 15, June 22-23

Live Demo: starting 100x Docker Containers on a Raspberry Pi 2 B

showed a screen shot with 250x running Containers

and plenty of resources/memory left, let’s improve it

OK, let’s start the Challenge

Challenge: How many Containers could be really started on a Pi 2?

my wish: I’d like to see 1000x Containers

personal record to this time: 613x Containers

--- but honestly, I was not sure, if this is possible at all!

let’s break the challenge

First tries to the recordTry, Learn and Share !

1 - Measure and automate all the things

Measures :

sysstat for post mortem

node-collector from Prometheus.io for “real time”

Provisionning :

Basic shell script published on my Github

Let’s try Ansible later : easier to maintain, need to learn :)

2 - Lower the container footprint

Tried with custom compiled nginx for ARM with few extensions ➢ ~ 80 containers

Footprint too big per container. Reading carefully Hypriot Blog : "rpi-nano-httpd" : 16 Kb footprint, 88 Kb image

➢ ~150 containers

3 - Docker tuning

Wanted to disable tcp-proxy process : no use here

Docker logs says “iptables” errors

Footprint with iptable + net. namespace : around 1 Mb !

Disable network / port forwarding + --net=host (Client side)

--bridge=none --iptables=false --ipv6=false --ip-forward=false --ip-masq=false -tls=false --userland-proxy=false (Engine side)

4 - Moaaar Docker tuning

No debug : --debug=true

No filesystem writing : --read-only=true

No logging : --log-driver=none

Following Docker upgrades :

--ipc=host

--utc=hostReached ~ 950 containers

5 - Linux tuning : start sharing

Tune sysctl : limits ! (MaxOpenFiles, MaxProcs, etc.)

Still stuck, footprint / container still big (~ 750 Kb)

Help from @booya and Hypriot guys + eading the /etc/init.d/docker script : Stack size (8 Mb -> 2 Mb in Hypriot:Jack)

Playing around doing abacus : <32 Kb breaks docker pull, <16Kb breaks docker run

~ 1700 containers

6 - To the record : Learning and sharing

GOGC to 10

~ 2 200 containers

Docker Hack days + Twitter :KSM : KO (need to recompile Kernel)

Learning the Linux memory management : Freeing caches + disabling OOM

~ 2498 containers !!!

Share and challenge !Community, discussions, challenge and Whales

BreizhCamp Hack (night) Party

Brainstorm for ideas

… then test everything in arbitrary order

help each other

drink some beer

learn a lot

Our Methodology

where to start?

first naïve tryonly 38 containers :-\ !

but 70 on a RPi1

figure out RPi2 limits without Dockerweb server footprint

network namespace footprint

get some help !let’s collaborate with @DamienDuportal (aka “French mafia”)

web server

from hypriot/rpi-nano-httpd image

written in ARM assembly code

already highly optimized1 page for code

1 page for data

1 page for stack

1 page for vsdo

=> 16kb memory footprint per process !

launched 27.000 on a RPi2

network namespace RPi2 limit

launched web server in a dedicated network namespace

ip netns exec <NS_NUMBER> httpd

RPi2 limit is ~ 1.100 network namespace

=> To break the challenge, we needed to run without network isolation

network configuration

disabled everything we could on Docker daemon

--bridge=none --userland-proxy=false --sig-proxy=false --ipv6=false --ip-masq=false --iptables=false

started containers with --net=host

disabled IPv6 on Linux kernel boot with ipv6.disable=1

Systemd

Docker daemon run as root

… but still has some limits set by systemd (so the 38 containers...)

LimitSIGPENDING=infinityLimitNOFILE=infinityLimitAS=infinityLimitNPROC=infinityLimitSTACK=?

stack size

Damien pro-tip : adjust LimitSTACK parameter to your needs

Default stack size is 8Mba stack consume 8Mb of process VM space (8 * 4 * 38 = 1,2 Gb)

=> ~ 1800 / 2000 containers

process memory layout

RPi2 processor is 32 bits !32 bit process address space

only 3Gb of virtual memory per process

collision between stack and heap memory areas

reduce thread stack sizesmallest working thread stack size: 24kb

reduce Golang memory consumptionlaunched docker with GODEBUG=gctrace=1 GOGC=1

speed up tests

launching thousands of containers on a RPi2 takes hours if not days!

everything in memory with zram devicesswap (ratio 5:1)

/var/lib/docker on ext4 FS (ratio 10:1)

swap as early as possible to keep free memory (vm.swappiness = 100)

USB external disk vs low perf, I/O limited SD card

more CPU for GO with GOMAXPROCS=4

reduce kernel perf event slowdown kernel.perf_cpu_time_max_percent = 1

system tuning

limit memory consumptionreduce GPU memory to 16Mb (can’t do less)

blacklisted non required Linux modules

remove some Linux limitsvm.overcommit = 1

kernel.pid_max = 32768

kernel.threads-max = 14812

Challenge completed!

We started 2499 containers !

RAM on RPi2 was not exhausted but Docker daemon crashed

docker[307]: runtime: program exceeds 10000-thread limit

Why is there a limit?

4 threads per container

10.000 threads for a Go application => 2500 containers max

Unlocked this with runtime.debug.SetMaxThread(12000)

hack not eligible for RpiDocker challenge, was just to confirm

can run ~2740 webserver containers, before actual OOM

what did not work

Btrfs

not working properly : strange web server 404 failures after ~20 successful launchs

stick with overlayfs

LXC driver

way sloooooooower

4 threads per container anyway

Go 1.5

compiled Docker with Go 1.5 for “better GC”, had no significant impact

What’s next

Understand why Docker do need 4 threads per container

(hey, lot’s of Docker core contributors here, time to ask !)

Collaboration (and beer) were the keys

to break this challenge !

links

Nicolas’ blog

http://blog.loof.fr/2015/10/how-to-run-2500-webservers-on-raspberry.html

code

https://github.com/ndeloof/RPiDockerKillerScore