ebpf based container networking - os3 · 2017-09-08 · ebpf based container networking a network...
TRANSCRIPT
![Page 1: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/1.jpg)
eBPF Based Container Networking
A Network Performance Comparison
Nick de Bruijn
July 4, 2017
University of Amsterdam
![Page 2: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/2.jpg)
Introduction
Figure 1: Microservices and Containers1
1https://www.slideshare.net/Docker/cilium-network-and-application-security-with-
bpf-and-xdp-thomas-graf-covalent-io
1/28
![Page 3: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/3.jpg)
Introduction - Iptables
Iptables:
• $ iptables -A INPUT -p tcp -s 10.0.0.23 –dport 80 -m conntrack
–ctstate NEW -j ACCEPT
2/28
![Page 4: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/4.jpg)
Research Goal
Research goal:
• Evaluate the usability of Cilium as a packet filtering system in a
container (Microservices) infrastructure.
3/28
![Page 5: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/5.jpg)
Research Questions
• What throughput and latency we get in the case of using Cilium’s
eBPF program and Linux’s Iptables as packet filter?
• What effect does the number of security policies have on the
throughput and latency in both cases?
• Is there a turn point in performance when increasing the number of
security policies?
4/28
![Page 6: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/6.jpg)
Background
5/28
![Page 7: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/7.jpg)
Docker Networking
• Endpoints (Container eth0)
• Virtual Ethernet devices (veth)
• Bridge on the host (docker0)
1Figure: https://success.docker.com/Architecture/DockerReferenceArchitecture
6/28
![Page 8: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/8.jpg)
Docker Networking - Communication
• Endpoints (Container eth0)
• Virtual Ethernet devices (veth)
• Bridge on the host (docker0)
Packet filtering:
• On container
7/28
![Page 9: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/9.jpg)
Docker Networking - Communication
Components:
• Endpoints (Container eth0)
• Virtual Ethernet devices (veth)
• Bridge on the host (docker0)
Packet filtering:
• On container
• On the bridge
8/28
![Page 10: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/10.jpg)
Iptables - Performance penalty?
• Uses chains with rules
• Each chain contains 0 or more
rules
• Top down approach
• Checks until match is found
• So placement is important
2
2Figure: http://www.iptables.info/en/structure-of-iptables.html
9/28
![Page 11: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/11.jpg)
What is Cilium?
• Opensource project
• Adds a layer on top of the
existing container environment
(Docker)
• To improve container
networking and policy
enforcement
• No Iptables / bridges
• Relies on eBPF programs
10/28
![Page 12: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/12.jpg)
What is eBPF (extended Berkeley Packet Filter)?
eBPF is used to extend the functionality of the kernel at runtime.
• It’s effectively a small kernel based machine
• 10 64bit registers
• 512 byte stack
• Data structures are known as maps
• Has a verifier to ensure the program is safe
• No loops, max 4k instructions, no more then 64 maps.
11/28
![Page 13: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/13.jpg)
eBPF
Figure 2: eBPF Overview3
3https://www.slideshare.net/Docker/cilium-bpf-xdp-for-containers-66969823 12/28
![Page 14: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/14.jpg)
extended Berkley Packet Filter - Functionality
1. Rewrite packet content
2. Extend/trim packet size
3. Redirect to other netdevices
4. Enforce policies
5. On the fly program generation
13/28
![Page 15: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/15.jpg)
Cilium - Network with eBPF
Figure 3: eBPF with Cilium4
4https://www.slideshare.net/Docker/cilium-bpf-xdp-for-containers-6696982314/28
![Page 16: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/16.jpg)
Cilium - Policies
Figure 4: Cilium Policy Using Json
15/28
![Page 17: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/17.jpg)
Approach
16/28
![Page 18: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/18.jpg)
Approach - Docker environment
17/28
![Page 19: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/19.jpg)
Approach - Cilium environment
18/28
![Page 20: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/20.jpg)
Approach - Scenario
Performed tests on two scenarios:
• Localhost
• And Multi-host
For each scenario we are interested in:
• The throughput and latency with no additional policies/rules.
• The change in performance whenever we start to increase the
number of policies/rules.
19/28
![Page 21: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/21.jpg)
Approach - Experiments
• Using Iperf3 to send a TCP STREAM
• Using Netperf to send a TCP RR (Request Response)
• Every test runs 1 minute. Every test is performed 10 times to
determine the variation
• Every test runs with 0, 1, 5, 10, 25, 50, 100, and 200 policies
20/28
![Page 22: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/22.jpg)
Results
21/28
![Page 23: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/23.jpg)
Results - Throughput Localhost
Figure 5: Throughput - localhost (Higher is better)
• Cilium’s eBPF approach outperforms the IPtable approach.
• Number of Cilium policies does not affect the throughput
• Number of no matching Iptables rules greatly affect the throughput 22/28
![Page 24: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/24.jpg)
Results - Latency Localhost
Figure 6: TCP Latency - localhost (Lower is better)
• Same observation as the throughput
• Cilium’s eBPF approach has a lower latency
23/28
![Page 25: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/25.jpg)
Results - Throughput Remote Containers
Figure 7: TCP Throughput - Remote Host (Higher is better)
• Different observation than on Localhost
• Cilium’s eBPF seems to perform less
• Iptables show no performs penalty until 1000 policies
24/28
![Page 26: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/26.jpg)
Results - Latency Remote Containers
Figure 8: TCP Latency - Remote Host (Lower is better)
• Same observation as the remote throughput
• Cilium’s eBPF approach has a higher latency
25/28
![Page 27: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/27.jpg)
Conclusion
26/28
![Page 28: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/28.jpg)
Conclusion
Overal:
1. Cilium seems like a promising project.
2. We can define L3, L4, and L7 policies
Performance wise:
1. The performance is not influenced by number of policies.
2. Cilium shows to perform better in the situation of local containers.
3. Room for improvements for multi-host enviornments
27/28
![Page 29: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/29.jpg)
Open issues & Future work
• Test the VXLAN overlay overhead used by Docker and Cilium
• Do Kernel traces to get a better understanding of which path
packets take in the kernel.
• Optimize both approaches to see what the best possible throughput
and latency can be reached for each approach.
• Test Cilium using XDP to offload the system.
28/28
![Page 30: eBPF Based Container Networking - OS3 · 2017-09-08 · eBPF Based Container Networking A Network Performance Comparison Nick de Bruijn July 4, 2017 University of Amsterdam](https://reader030.vdocuments.site/reader030/viewer/2022041014/5ec557b313b08355f20a9fb1/html5/thumbnails/30.jpg)
Thank you for your attention,
Questions?
28/28