Download - Networking in Docker Containers
Page 1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking in ContainersAttila Kanto
Page 2 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Agenda
• How networking works in Docker
• Container Network Model
• Networking plugin
Page 3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Containers
• Isolate and package applications• Resources (CPU, memory, IO)• Namespaces (pid, users, network, uts, mnt )• Storage (device mapper, overlayfs, aufs, btrfs)• Security (capabilities)
Page 4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Network
• UTS namespace• isolate hostname
• Network namespace• network interface(s)• loopback device• routing table• iptable rules
Page 5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Basic networking overview
5
Page 6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking without Docker
eth0
iptables
route
Page 7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking without Docker
ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255
ether 33:83:5a:44:50:ff txqueuelen 0 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
Page 8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking without Docker
ifconfig
eth0:
inet 192.168.1.100
ether 33:83:5a:44:50:ff
OSI Layers (1 – 4)
Page 9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking without Docker
route -n
Destination Gateway Genmask Iface
0.0.0.0 192.168.1.1 0.0.0.0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 eth0
iptables -t nat -L
target prot opt source destination
Page 10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking without Docker
eth0
iptables
route192.168.1.0/24 -> eth00.0.0.0 -> 192.168.1.1 (eth0)
192.168.1.100
Page 11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Networking with Docker
11
Page 12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Install Docker
eth0
iptablesMASQUERADE 172.17.0.0/16
route192.168.1.0/24 -> eth00.0.0.0 -> 192.168.1.1 (eth0)172.17.0.0/16 -> docker0
192.168.1.100 172.17.0.1
docker0
Page 13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Run container / bridged networking
• Docker0 bridge• already there, created during install
• Network namespace• container netns needs to be created
• Veth pair• created during the creation of container• connects two network namespaces
• External communication• Only through Network Address Translation (NAT)
Page 14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Run container / bridged networking / 8080 -> 9090
eth0
iptablesMASQUERADE 172.17.0.0/16DNAT dpt:9090 to:172.17.0.2:8080
route192.168.1.0/24 -> eth00.0.0.0 -> 192.168.1.1 (eth0)172.17.0.0/16 -> docker0
192.168.1.100 172.17.0.1
docker0
container1ns
eth0vxxveth
172.17.0.2
route
SRC DSTClient Port 9090Client IP 192.168.1.100Client MAC MAC of eth0
SRC DSTClient Port 8080Client IP 172.17.0.2
SRC DSTClient Port 8080Client IP 172.17.0.2
MAC of docker0 MAC of eth0
Page 15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Overlay networking with Docker
15
Page 16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Run container / overlay networking• Bridges
• docker_gwbridge created if does not exist• br0 in a “hidden” namespace associated with the overlay network
• Network namespace• container netns needs to be created
• Veth pairs• connects br0 and and eth0 of container• connects docker_gwbridge and eth1 of container
• External communication• Through Network Address Translation (NAT)• Through VXLAN (other container using the same overlay network)
Page 17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Install Docker (again)
eth0
iptablesMASQUERADE 172.17.0.0/16
route192.168.1.0/24 -> eth00.0.0.0 -> 192.168.1.1 (eth0)172.17.0.0/16 -> docker0
192.168.1.100 172.17.0.1
docker0
Page 18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Run container / overlay networking
eth0
iptables
route
192.168.1.100
172.18.0.1
docker_gw
container1ns
eth1vxx veth172.18.0.2
172.17.0.1
docker0
ns
br0 eth0vyyveth
10.10.10.210.10.10.1
VXLAN
route
Page 19 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Software-defined networking (SDN)
• Separation control and data plane of network
• Control plane• makes decisions about where traffic is sent
• Data plane• forward traffic to the selected destination
Page 20 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Data Plane (in Docker overlay)
• Virtual Extensible LAN (VXLAN)• overlay technology• encapsulates L2 frames as UDP packets
• VTEP – VXLAN Tunnel End Point• originator and/or terminator of VXLAN tunnel
• VNI – VXLAN Network Identifier• part of the VXLAN Header• similar to VLAN ID
Page 21 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Data Plane (in Docker overlay)
• Container sends a packet• ARP (neighbor) table is checked for destination container IP -> MAC
interface mapping• L2 FDB (forwarding database) is checked to determine IP of destination
VTEP for destination MAC on source VTEP• packet is encapsulated for destination VTEP with configured VNI and sent
to destination• destination VTEP de-capsulates the packet • inner packet is received by the destination container
Page 22 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Network Control Plane (in Docker overlay)
Page 23 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Container Network Model
23
Page 24 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Container Network Model (CNM)• Sandbox
• holds the config of a container's network stack (DNS, routing, etc.)• multiple endpoints from multiple networks• Linux Network Namespace / FreeBSD Jail
• Network• Group Endpoints that are able to communicate with each-other directly• Linux Bridge / VXLAN
• Endpoint• joins Sandbox to Network• veth pair / ovs patch port
Page 25 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Docker libnetwork
• Docker’s networking library• Implements CNM• Built-in drivers (in process)• Network drivers (bridge, overlay)• IPAM drivers
• Plugin mechanism (off process)• External Network drivers (Calico, Midonet, my own driver) • External IPAM drivers
Page 26 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Libnetwork plugins
• Implemented using libnetwork’s remote driver• Running off-process (not in Docker daemon)• HTTP POSTs with JSON payload• KV store API not exposed• can be implemented in any programming language
• KV store• KV url / credentials needs to be passed in init time
• Can be deployed as container
Page 27 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Network plugin API (Network)
• CreateNetwork
• DeleteNetwork
Page 28 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Network plugin API (Endpoint)
• CreateEndpoint
• DeleteEndpoint
Page 29 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Network plugin API (Join)
• Join
• Join (resp)
Page 30 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Floating IP network driver
• Containers on same L2 network• Connected with Open vSwitch• IP Address Management• libnetwork built-in IPAM driver is used
• Externally addressable IP / container• no Network Address Translation• no port collision• extremely fast• scalability
Page 31 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Run container / floating driver
iptables
route
192.168.1.100
container1ns
172.17.0.1
docker0
floating_bridge
eth0
192.168.10.2
eth1 veth2veth1
container2ns
eth0
192.168.10.3
veth veth
eth0
Page 32 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Demo
32
Page 33 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
How to use it in Hadoop world
• Using multiple networks• overlay to create internal network• floating for exposing servers
Data Node
Data Node
Data Node
Ambari
Master Node
Data Node
Data Node
Data Node
Master Node
Edge Node
OverlayFloating
Page 34 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Takeaways
• Since 1.9 Docker networking has improved• Easy to write a plugin that does certain things better• Multiple networks can be used by the same container• Not everybody is happy with it
• Kubernetes http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-libnetwork.html• Mesos https://issues.apache.org/jira/browse/MESOS-3828
Page 35 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
We are hiring!
35