Modular Overlay Networking Solutions with the Container Network Interface

Murali PaluruPrincipal Software Engineer,

Rancher Labs, Inc.

Open Source Summit North America,13th September 2017

Can you spot the bugs?

Answers

Slides (without animation) are posted in PDF format here:https://ossna2017.sched.com/event/BDpo/modular-overlay-networking-solutions-with-the-container-network-interface-murali-paluru-rancher-labs-inc

Original Slides (with animation) can be found here:https://goo.gl/L2bdZt

Slides availability

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

Who am I?● Plumber (Principal Software Engineer) at Rancher Labs.

Build the pipes between containers (Take care of networking)

What have I done?https://github.com/leodotcloud

- Build, maintain networking microservices for Rancher product (complete container management platform)- CNI plugins: VXLAN, IPSec- Network Policy- Solve networking issues in customer setups

Past: Startup, Gigamon, Cisco

How to reach out to us/me?Twitter: @leodotcloud

Email: mpaluru@rancher.com or leodotcloud@gmail.com

Rancher Product: https://github.com/rancher/rancher

Community Users: https://forums.rancher.com

Slack: https://slack.rancher.io

If you want to play with containers/microservices: https://try.rancher.com

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

Docker Networking - Single Host/Server/Node

HOST 1

Who is responsible for setting up networking?

HOST 1

Docker Networking made easy

Subnet 172.17.0.0/16: {172.17.0.1, 172.17.0.2, … } { Amy, Bob, Deb, Eli, Kim …. }

HOST 1

Containers in the same host can talk to one another

Bob

Deb

AmyHOST 1

What if we add more servers?

HOST 1 HOST 2

Can containers can talk?

HOST 1 HOST 2

Bob

Deb

Amy

Bob

Deb

Amy

Can containers can talk?

HOST 1 HOST 2

Bob

Deb

Amy

Bob

Deb

Amy

Confusion!Chaos!!!

One of the solutions: Published ports

10.250.0.100:300010.250.0.100:32769...

10.250.0.200:808010.250.0.200:32769...

Published ports (made easy)

10.250.0.100:300010.250.0.100:32769...

10.250.0.200:808010.250.0.200:32769...

Bob

Deb

Amy

Bob

Deb

Amy

Floor 1

Floor 2

Floor 1

Floor 2

Building Alpha Building Bravo

Published ports: Containers on different servers can talk!

10.250.0.100:300010.250.0.100:32769...

10.250.0.200:808010.250.0.200:32769...

Bob

Deb

Amy

Bob

Deb

Amy

Floor 1

Floor 2

Floor 1

Floor 2

Building Alpha Building Bravo

Hi, this is the person calling from second floor of Building Bravo, can I talk to to the person living on first floor of Building Alpha?

Rancher’s solution: Unique subnet for containers on all hosts

Rancher’s solution: Unique subnet for containers on all hosts

Subnet 172.17.0.0/16: {172.17.0.1, 172.17.0.2, … } { Amy, Bob, Deb, Eli, Kim …. }Subnet 10.42.0.0/16: {10.42.0.1, 10.42.0.2, …} { Gandhi, Ram, Budh, Vyas, Meera …. }

Rancher’s solution: Unique subnet for containers on all hosts

Bob (Ram)

Deb (Budh)

Amy (Gandhi)

Bob (Vyas)

Deb (Meera)

Amy (Gandhi)

Kubernetes Solution

Kubernetes Solution (made easy)

Homework for your imagination

Subnet 10.10.1.0/24: {10.10.1.1, 10.10.1.2, … } { ?, ?, ? …. }Subnet 10.10.2.0/24: {10.10.2.1, 10.10.2.2, … } { ?, ?, ? …. }

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

How do I plug in?

Industry wants a standardA standard/interface/API is required so that 3rd party vendors can plug in their custom network solution into a given container runtime.

Docker launches CNM (libnetwork)https://www.sdxcentral.com/articles/news/docker-acquires-networking-startup-socketplane/2015/03/

CNM: Container Network Model

Source: https://github.com/docker/libnetwork/blob/master/docs/design.md

CoreOS(rkt) releases CNI as part of appcCNI: Container Network Interface

Organisation for the App Container specification, including the schema and associated tooling: https://github.com/appc

Part of effort to define open standards/specifications to support other runtimes.

CNM vs CNIWorks with Docker runtime.

Daemon

Project Calico, Weave, Cisco Contiv, Kuryr, Open Virtual Networking (OVN), VMware, …

Needs a KV store for global plugins.

Built in: none, bridge, macvlan, ipvlan, etc.

Support for multiple networks.

Works with rkt, kurma, …

Binary

Kubernetes, Rancher, Project Calico, Weave, Cisco Contiv, Apache Mesos, Cloud Foundry, Kurma, Cilium, Infoblox, …

It’s up to the plugin to decide.

Built in: loopback, bridge, macvlan, ipvlan, etc.

Multiple networks: Work in progress

Disadvantages:Some problems with docker ps and docker inspect

CNI adoptionKubernetes(k8s) adopted CNI and blogged about the reasons behind their decision: http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-libnetwork.html

Project Calico.

Weave.

Mesos embraces CNI.

Rancher embraces CNI.

CNCF accepted CNI. https://www.cncf.io/blog/2017/05/23/cncf-hosts-container-networking-interface-cni/

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

What are the responsibilities of “network plugin”?

1. Take care of plumbing, plugging a container in a network, provide connectivity, etc.

2. Also need to give an identity (IP, etc, etc) to the container.○ Sometimes, the plugin can do this myself○ In general the plugin needs help and ask another plugin (IPAM plugin) for this information.

Let’s get started: Pick a runtime to run containers

runtime

Step 0: Setup CNI config file

runtime

Step 1: Runtime adds/deletes a container

runtime container1

Step 1: Runtime adds/deletes a container

runtime container1

NOTE: In case of using docker, the container is started with “--net=none”

Step 2: Run time invokes CNI network plugin

runtime container

CNI network plugin

1

2

Step 3: CNI network plugin invokes IPAM plugin (optional)

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

Step 4: IPAM plugin allocates/deallocates, returns IPAM info (optional)

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

Step 5: CNI network plugin setup/cleanup networking for container

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

5

Step 6: CNI plugin informs runtime about success/failure

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

56

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

JSON => Step 0: Setup CNI config file

runtime

JSON => Step 2: Run time invokes CNI network plugin

runtime container

CNI network plugin

1

2

●●

CNI_VERSIONCNI_CONTAINERIDCNI_NETNSCNI_IFNAMECNI_COMMANDCNI_PATHCNI_ARGS

JSON => Step 3: CNI network plugin invokes IPAM plugin (optional)

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

●● CNI_VERSION

CNI_CONTAINERIDCNI_NETNSCNI_IFNAMECNI_COMMANDCNI_PATHCNI_ARGS

JSON => Step 4: IPAM plugin allocates/deallocates, returns IPAM info (optional)

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

●●

JSON => Step 4: IPAM plugin allocates/deallocates, returns IPAM info (optional)

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

●●

JSON => Step 6: CNI plugin informs runtime about success/failure

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

56

●●

JSON => Step 6: CNI plugin informs runtime about success/failure

runtime container

CNI network plugin

1

2

CNI IPAM plugin3

4

56

●●

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

CNI Github Repositories● https://github.com/containernetworking/cni

○ contains the Go source code of a library for integrating CNI into applications (libcni)○ example command-line tool (cnitool)

● https://github.com/containernetworking/plugins○ Main: bridge, loopback, ipvlan, macvlan, ptp, vlan○ IPAM: host-local, dhcp○ Meta: tuning, portmap, flannel

The above plugins can be forked and customized to your needs.Example: rancher-cni-bridge, rancher-cni-ipam

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

Files for demo

https://github.com/leodotcloud/ossna2017-cni-presentation

Agenda● Introduction● Evolution of Container Networking● History of Container Network Interface (CNI)● CNI Overview● Walkthrough of CNI Specification● SDK + Reference CNI plugins● Build a simple CNI runtime + CNI plugin● Questions (You don’t have to wait till the end, interrupt!)

Questions?

Thank you!