kernel load-balancing for docker containers using ipvs
TRANSCRIPT
IPVS for Docker Containers
Andrey SibiryovBulldozer Operator at Uber
This guy on the stage
2
…who the hell are you and why should we believe a Russian?
What is IPVS…and why didn’t we hear about it before?
International Pig Veterinary Society
4
IP Virtual Server
5
IP Virtual Server
6
…and why didn’t we hear about it before?
IP Virtual Server
7
Works inside the Linux Kernel, based on Netfilter.Supports TCP, SCTP & UDP, v4 and v6.8+ methods: WRR, WLC, LBLCR, SH and much more – plugins.NAT, Tunneling, Direct Routing.Address bundling via FWMark services.Most importantly, it is bloody fucking fast!
IP Virtual Server
8
…and a little bit more about these weird acronyms.
IPIPEncapsulates IP
Routable anywhere
NATRewrites DST IP
Same L4
DSRRewrites DST MAC
Same L2
IP Virtual Server
9
…and a few more words about DR since it’s awesome!
TPS
50
66
82
98
114
130
Single IPVS DR HAProxy MySQL Proxy
I don’t need this
10
…and why would we load balance and route anything at all?
I don’t need this
11
…also, our nginx, haproxy, hipache, vulcand setup works fine – get off the stage, please!
I don’t need this
12
…and we run our stuff in the cloud, it takes care of everything – our work is perpetual siesta!
IP Virtual Server
13
IPVS NGINX HAPROXY AWS ELB
OSI layer L4 L7 L7 L4TCP ✅ ℹ ✅ ✅
UDP ✅ 🚫 🚫 🚫
Dynamic configuration ✅ 🚫 ℹ ✅
Forwarding methods 4 NAT NAT NATBalancing methods 8+ 3 6 RR
Health checks ✅ 🚫 ✅ ✅
What is IPVS…and how do we use it since it sounds so amazing!
What is GORB
16
…and why is it cool for Docker Containers?
17
REST API daemon to control IPVS.Simple GET, PUT, DELETE interface.Docker Link to control GORB.TCP, HTTP health checks built-in.Written in Go, LGPL licensed, on GitHub.Docker + GORB + MacVLAN network plugin + IPVS DR forwarding mode = ❤
What is GORB…and why is it cool for Docker Containers?
What is GORB…and how do we use it? Live demo or GTFO!
What is GORB
19
The guy on the stage is doing roughly this right now.
kobolog@gorb:~$ go install github.com/kobolog/gorb
kobolog@gorb:~$ sudo modprobe ip_vs
kobolog@gorb:~$ sudo gorb &
kobolog@docker:~$ go install github.com/kobolog/gorb/gorb-docker-link
kobolog@docker:~$ export DOCKER_HOST=<...>
kobolog@docker:~$ gorb-docker-link -r gorb:4672 &
kobolog@docker:~$ docker run -d -p 80 nginx
kobolog@gorb:~$ curl -i -XGET http://localhost:80
A few more words about BGP
20
…black belt in networking is not complete without a few words about BGP.
How stable is GORB
21
…and is it production-ready? Can we blame you if it doesn’t work?
«I'll let you in on a secret: my pet hamster did all the coding. I was just a channel, a «front» if you will, in my pet's grand plan. So, don't blame me if there are bugs. Blame the cute, furry one» —Rusty Russel
22
€25,000
23
…is the price for an enterprise hardware load-balancer.
€0
24
…is a price for IPVS and BGP (and GORB).
Works on commodity hardware.No special software or tools, no SNMP.1000-word man page vs. 1000-page vendor manual.1 GBPS line rate = 1% CPU in DR mode.IPVS is around for more than 15 years.
25
A good way to spend €25,000
