kubernetes on open stack
TRANSCRIPT
![Page 1: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/1.jpg)
Kubernetes on OpenStack
ByNaveen Joy
Cloud Architect
![Page 2: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/2.jpg)
Why Kubernetes?Provides a higher level abstraction to a lower level docker interfaceOrganize applications running in docker containers into PODs
PODs form the basic unit of operation POD == set{ one or more containers }
Users declare end state using a POD manifestScheduling mechanism for PODsContainers in a POD are tightly coupled i.e. co-located on a host and share network namespace, volumes and hostname
![Page 3: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/3.jpg)
Why Kubernetes?Ability to group PODs using labelsEnable access to the POD group using a service abstraction (provides a stable service VIP)
The service will keep track of its PODs - endpoints of a serviceWhen traffic hits the service virtual IP, it will be proxied to one of the backend PODs
POD ManagementRestart a failed container in a POD automaticallySelf healing - ability to replace PODs when the machine failsHorizontal scaling
![Page 4: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/4.jpg)
Architecture Overview
![Page 5: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/5.jpg)
Networking Model
![Page 6: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/6.jpg)
Default Networking Model in Docker• Host-Private Networking• Creates a virtual bridge named docker0 on each host• Allocates a private subnet (e.g. 172.17.0.0/16) from RFC 1918
for that bridge • Attaches each container to docker0 using a virtual ethernet
device • Assigns an IP from the private subnet to the container and sets
the bridge IP address is set as the gateway for the container
![Page 7: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/7.jpg)
Default Docker Networking Model
![Page 8: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/8.jpg)
Container reachability across hostsDocker may allocate the same IP addresses to containers across hosts
Containers can talk to each other on the same machine Containers cannot route traffic directly across hosts using their private IP address
Containers communicate across hosts by using DNAT
Host IP:Port To Container IP:Port
![Page 9: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/9.jpg)
Default Networking model in docker can pose issues to Apps
Coordinating static port allocations to containers is very difficult in practice across multiple developers and groups that share hostsIf using dynamic port allocation, there are still complications
service discovery, application configuration etc.
NAT is hard to troubleshootApplication running in a container does not know its actual IP address – so some apps will break
apps that need to register their actual IP addressapps that perform IP based access control/authentication
![Page 10: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/10.jpg)
Networking in Kubernetes
Containers communicate directly over a routed IP network without using NAT
A container sees the real IP of another containerThe host sees the real IP of the container
The default networking model of docker must be modified for Kubernetes to work
![Page 11: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/11.jpg)
Networking in Kubernetes
• A routable IP address is assigned per POD
• All containers within a POD share the network namespace including the IP address and port• Implemented by creating a docker container
for the POD• This “pod-container” is wired to the POD IP• All other containers are configured to share
the network stack of the POD container using the --net=container:<name | id > function in docker
![Page 12: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/12.jpg)
POD networking
• Each VM is assigned a subnet for POD networking (Note: This is in addition to the main neutron subnet used by the VM)
• The default docker bridge docker0 is replaced with a linux bridge say “cbr0”
• cbr0 is configured on the POD subnet• Docker daemon is started with this
bridge using --bridge=cbr0 in its options
• Docker allocates IPs to the containers from the POD subnet block
![Page 13: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/13.jpg)
Routing POD SubnetsOption 1:
Create routable POD networks1. Configure instances to forward IP packets to the bridged POD
network by enabling IP forwarding in the kernel • sudo sysctl net.ipv4.ip_forward = 1
2. Add static routes on the L3 neutron gateway to route traffic to the instance• neutron router-update --routes type=dict list=true \
destination=NODE_X_POD_CIDR, \nexthop=NODE_X_INTERFACE_IP_ADDR
![Page 14: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/14.jpg)
Routing POD SubnetsOption 1 :
3. When neutron security-groups is enabled, traffic is restricted to/from the instance IP address by neutron
• Add iptables FORWARD chain rules on the host to allow incoming and outgoing traffic to/from the POD CIDR
POD_CIDR=10.5.0.0/16
sudo iptables -I FORWARD 1 -p all -s $POD_CIDR –d $POD_CIDR -j ACCEPT
![Page 15: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/15.jpg)
Option 1: Network Diagram
![Page 16: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/16.jpg)
Routing POD SubnetsOption2:Build an overlay network to route POD networks• Proceed with caution for production deployment • These technologies are still in experimental stage
• Creates a layered virtual network architecture • Create POD virtual network overlay using the neutron virtual
networks as the underlay
• Open source options: • Flannel, Weave, Calico
![Page 17: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/17.jpg)
Flannel• Designed for Kubernetes• Creates a POD subnet on each instance• Uses etcd to maintain the subnet to real host IP mapping• Builds an overlay mesh network between instances using
UDP tunneling to connect the subnets• Requires UDP port 8285 opened in the instance security
groups• Adjust the MTU size for performance
![Page 18: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/18.jpg)
Option 2: Network Diagram
![Page 19: Kubernetes on open stack](https://reader035.vdocuments.site/reader035/viewer/2022062821/5888ad781a28ab80248b5279/html5/thumbnails/19.jpg)
ConclusionCheckout the Kubernetes github repo
Latest docsContains several deployment examplesSaltStack scripts to automate a cluster deployment across multiple providers