docker, atomic host and kubernetes

Docker, Atomic Host and Kubernetes.Red HatJooho Lee Technical Support Engineer

IT Trend - Cloud• Cloud Computing Trends (Right Scale)

What is DevOps?• DevOps (Developers + Operations)

• Software development methodology• Culture• Emphasize on communication, collaboration and integration.• Achieve rapid release.

Devel-oper

Opera-tions

Rapid Change StabilitySeparate workflowDifferent object

Different situation

Key points in DevOps• Worth & Object

• Respect• Share value• Share ownership• Agreement

• Process• Share same workflow• Synchronize focus • Decrease cycle time

• Tool• Automation

• Package• Build• Test

Operating Sys-tem

InfrastructureApplication

Platform

QA

De-vOps

DevOps is not only development culture but also collaboration process,which could develop and operate service independently in a organization

DevOps• Benefit

• Innovate faster• More responsive to business needs• Better collaboration• Better quality• More frequent releases

• Necessary• New Mindset (Open mind)• New Tools (Provisioning, Monitoring, etc)• New Skills (From platform to QA)

Docker Concept• Share kernel (Lightweight)• Share resources (Fast)• Share image (Portable)• Layered file system

(Easy commit)• …….

Why DevOps prefer docker?• Container-based Platform• Easy/Fast deployment, build, provisioning• Similar performance compared to bare-metal• Low learning curve (Dockerfile use bash shell)

• Ansible(Python)/ Chef(Ruby and Erlang) / Puppet(Ruby) • RedHat, IBM, Microsoft, CoreOS... Many companies support.

DevOps needs more…

• Reliable system management. (Atomic Host, etc)• Handle network complexity (OVS, flanneld, etc)• Management module for bigger infrastructure. (Kubernetes, etc)

Docker in realMinimal Docker

What is Atomic Host?• Trusted operating system platform

• Container-based application• Service deployment

• End-to-End hosting architecture that’s modern, reliable and secure.

Reliable distribution OS High Secure Module SELinux

Container Management

Atomic Host Feature• Support packages

• Docker• Flannel• Kubernetes• rpm-ostree

• Support various file system for Docker• vfs• devicemapper• btrfs• aufs (not recommend for production, not supported)

• Networking• Single-host networking : Docker• Multi-host networking : Kubernetes & Flannel

How to debug Atomic Host ?• Atomic Host is minimal of RHEL system.• Does NOT use “YUM” on Atomic Host.• How debug or install packages?• Red hat provide “Red Hat Enterprise Linux Atomic Tools Container Image”

• It’s Big (1G)• Contains man pages• Opens privileges• May behave differently

• How?# docker pull rhel7/rhel-tools# atomic run rhel7/rhel-tools [root@localhost /]#

What is Kubernetes?• Greek for “pilot” or “helmsman of a ship”

• Kubernetes is an open source Container Cluster orchestration framework that was started by Google in 2014.

Kubernetes do WHAT?• Manage docker containers centrally.• Manage nodes.• Handle complex networking. …..

Manage a cluster of Linux containers as a single sys-tem

to accelerate Dev and simplify Ops

Kubernetes ArchitectureKubectl(user commands)

Schedul-ing

actuatorREST

(pods,services, rep.controllers)

Replication Controller

Stor-age

(etcd)

kubelet Proxy

……AuthAPIServer

Scheduler

Node

cadvi-sor

Pod Pod Pod

kubelet Proxy

……

Node

cadvi-sor

Pod Pod Pod

Main Terms• Master

• Managing machine, which oversees one or more minions.• Minion

• A slave that runs tasks as delegated by the user and Kubernetes master.• Pod

• An application (or part of an application) that runs on a minion. • Replication Controller

• Ensures that the requested number of pods are running on minions at all times.• Label

• An arbitrary key/value pair that the Replication Controller uses for service discovery• Service

• An endpoint that provides load balancing across a replicated group of pods• kubectl

• The command line config tool

Main Components • API Server

• REST server• Controller Tower

• Controller Manager• Replication Controller Management (Watches etcd)

• Scheduler• Communicate with minions• Decide a minion to distribute workload • Check if the task happen.

• Kubelet• Manage container deployments• Ensure the state of containers (which is supposed to be in)

• Kube-proxy• Route and forward traffic to and from containers

• ETCD• Distributed, consistent key value store for shared configuration and service discovery

How to configure Kubernetes?

# Comma seperated list of nodes in the etcd clusterKUBE_ETCD_SERVERS="--etcd_servers=http://192.168.20.10:4001"

configure

# The address on the local server to listen to.KUBE_API_ADDRESS="--address=0.0.0.0"...# How the replication controller and scheduler find the kube-apiserverKUBE_MASTER="--master=192.168.20.10:8080"

apiserver# Comma seperated list of minionsKUBELET_ADDRESSES="--machines=minion1,minion2,minion3,minion4"

controller-manager

Service Register# systemctl enable etcd kube-apiserver kube-controller-manager kube-schedulerService Start# systemctl start etcd kube-apiserver kube-controller-manager kube-schedulerService Check# systemctl status etcd kube-apiserver kube-controller-manager kube-scheduler# systemctl status etcd kube-apiserver kube-controller-manager kube-scheduler | grep active |wc –l4

Master : /etc/kubernetes/

How to configure Kubernetes?Master

{ "Network": "172.16.0.0/12", "SubnetLen": 24, "Backend": { "Type": "vxlan" } }

Create Flanneld-conf.json

Flanneld Configuration

curl -L http://localhost:4001/v2/keys/atomic01/network/config -XPUT --data-urlencode [email protected]

Create key/value on ETCD

http://localhost:4001/v2/keys/atomic01/network/config

How to configure Kubernetes?Minion

Create /etc/sysconfig/flanneld# etcd url location. Point this to the server where etcd runs FLANNEL_ETCD="http://192.168.122.10:4001" # etcd config key. This is the configuration key that flannel queries # For address range assignmentFLANNEL_ETCD_KEY="/atomic01/network"

[Unit] After=flanneld.service Requires=flanneld.service [Service] EnvironmentFile=/run/flannel/subnet.env ExecStartPre=-/usr/sbin/ip link del docker0 ExecStart=/usr/bin/docker -d --bip=${FLANNEL_SUBNET} --mtu=${FLAN-NEL_MTU} $OPTIONS $DOCKER_STORAGE_OPTIONS

Create /etc/systemd/system/docker.service.d/10-flanneld-net-work.conf

Flanneld Configuration

http://192.168.122.10:4001/

http://192.168.122.10:4001/

http://192.168.122.10:4001/

How to configure Kubernetes?

# Comma seperated list of nodes in the etcd clusterKUBE_ETCD_SERVERS="--etcd_servers=http://192.168.20.10:4001“

config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)KUBELET_ADDRESS="--address=192.168.20.11"...# You may leave this blank to use the actual hostnameKUBELET_HOSTNAME="--hostname_override=minion1"

kubelet

Minion

What does Flanneld do?

ens3192.168.10.

11

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4

Atomic HostPod A Pod B

Apache con-tainer

Apache con-tainer Apache container in Pod A

Apache container in Pod B

Communication ??


ens3192.168.10.

11

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4


Apache con-tainer

Apache con-tainer Apache container in Pod A

Apache container in Pod B

Communication OKthrough docker0 bridge


ens3192.168.10.

11

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4


Apache con-tainer

Apache con-tainer

ens3192.168.10.

12

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4

Atomic HostPod C Pod D

Apache con-tainer

Apache con-tainer

br0192.168.10.

1


ens3192.168.10.

11

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4


Apache con-tainer

Apache con-tainer

ens3192.168.10.

12

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4


Apache con-tainer

Apache con-tainer

br0192.168.10.

1

Apache container in Pod A Apache container in Pod CCommunication ??


ens3192.168.10.

11

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4


Apache con-tainer

Apache con-tainer

ens3192.168.10.

12

docker0172.17.42.1/

16

veth0172.17.0.1/2

4

veth0172.17.0.2/2

4


Apache con-tainer

Apache con-tainer

br0192.168.10.

1

Apache container in Pod A Apache container in Pod CCommunication NO


ens3192.168.10.

11

docker0172.16.32.1/2

4

veth0172.16.32.2/

24

veth0172.16.32.3/

24


Apache con-tainer

Apache con-tainer

br0192.168.10.

1

Flannel.1172.16.32.0/1

6flanneld

ens3192.168.10.

12

docker0172.16.10.1/2

4

veth0172.16.10.2/

24

veth0172.16.10.3/

24


Apache con-tainer

Apache con-tainer

Flannel.1172.16.10.0/1

6flanneld

Apache container in Pod A Apache container in Pod CCommunication YES

Label

Service Pod

Pod Pod



Replica-tion Con-

trollerService

Pod


Replica-tion Con-

trollerService

Label

Service Pod

Pod Pod



Replica-tion Con-

trollerService

Pod


Replica-tion Con-

trollerService

labels : name : apache

Label

Service Pod

Pod Pod



Replica-tion Con-

trollerService

Pod


Replica-tion Con-

trollerService

labels : name : apache port : 80

Podid: "apache"kind: "Pod"apiVersion: "v1beta1"desiredState: manifest: version: "v1beta1" containers: - name: "apache80" image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 cpu: 100labels: name: "apache" port: "80"

Pod

id: "apache"kind: "Pod"apiVersion: "v1beta1"desiredState: manifest: version: "v1beta1" containers: - name: "apache80" image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 cpu: 100labels: name: "apache" port: "80"

POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUSapache 172.16.69.238 apache80 local-registry:5000/rhel7-ews21:demo minion4/ name=apache,port=80 Running

Pod



min-ion1

min-ion2

min-ion3

min-ion4

Pod : apache

apache80

Container172.16.69.238

Pod



min-ion1

min-ion2

min-ion3

min-ion4

Pod : apache

apache80


172.16.69.28

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES1a0e11d8363e local-registry:5000/rhel7-ews21:demo "/bin/sh -c '$EWS_HO 14 minutes ago Up 14 minutes k8s_apache80.afee6806

Pod

id: "apache"….. containers: - name: "apache80" image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 cpu: 100 - name: "apache8080" image: local-registry:5000/rhel7-ews21:8080 imagePullPolicy: PullAlways ports: - containerPort: 8080 hostPort: 8080 cpu: 100labels: name: "apache"

POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUSapache 172.16.15.9 apache80 local-registry:5000/rhel7-ews21:demo minion3/ name=apache Running apache8080 jonlangemak/docker:web_container_8080

minion3

Pod : apache

apache80

172.16.15.9

apache8080

pause

Pod

id: "apache"….. containers: - name: "apache80" image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 cpu: 100 - name: "apache8080" image: jonlangemak/docker:web_container_8080 imagePullPolicy: PullAlways ports: - containerPort: 8080 hostPort: 8080 cpu: 100labels: name: "apache"

minion3

Pod : apache

apache80

172.16.15.9

apache8080

pause

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESaf1e66870aa1 jonlangemak/docker:web_container_8080 "/usr/bin/supervisor 7 minutes ago Up 7 minutes k8s_apache8080…..f3c0c5009a1b local-registry:5000/rhel7-ews21:demo "/bin/sh -c '$EWS_HO 10 minutes ago Up 10 minutes k8s_apache80.32….605fccc3e06f kubernetes/pause:latest "/pause" 10 minutes ago Up 10 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:8080->8080/tcp k8s_net.13b8d9d0_...

Replication Controllerid: apache-rc-controllerapiVersion: v1beta1kind: ReplicationControllerdesiredState: replicas: 3 replicaSelector: name: apache podTemplate: desiredState: manifest: version: v1beta1 id: apache-pod containers: - name: apache-pod image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 labels: name: apache port : 80

Replication Controllerid: apache-rc-controllerapiVersion: v1beta1kind: ReplicationControllerdesiredState: replicas: 3 replicaSelector: name: apache.… id: apache-pod containers: - name: apache-pod image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 labels: name: apache port : 80

CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICASapache-rc-controller apache-pod local-registry:5000/rhel7-ews21:demo name=apache 3

min-ion1

min-ion2

min-ion3

min-ion4

Pod : apache

apache80


POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUSff433d37-…-52540098677f 172.16.31.28 apache-pod loal-registry:5000/rhel7-ews21:demo minion1/ name=apache Running26a89e0e-…-52540098677f 172.16.20.15 apache-pod local-registry:5000/rhel7-ews21:demo minion2/ name=apache Runningd7deb52e-…-52540098677f 172.16.69.239 apache-pod local-registry:5000/rhel7-ews21:demo minion4/ name=apache Running

Pod : apache

apache80


Pod : apache

apache80



Replication Controllerid: apache-rc-controllerapiVersion: v1beta1kind: ReplicationControllerdesiredState: replicas: 1 replicaSelector: name: apache.… id: apache-pod containers: - name: apache-pod image: local-registry:5000/rhel7-ews21:demo imagePullPolicy: PullAlways ports: - containerPort: 80 hostPort: 80 labels: name: apache port : 80


POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUSd7deb52e-…-52540098677f 172.16.69.239 apache-pod local-registry:5000/rhel7-ews21:demo minion4/ name=apache Running

min-ion1

min-ion2

min-ion3

min-ion4

Pod : apache

apache80


Pod : apache

apache80


Pod : apache

apache80



Service

POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUSff433d37-…-52540098677f 172.16.31.28 apache-pod local-registry:5000/rhel7-ews21:demo minion1/ name=apache Running26a89e0e-…-52540098677f 172.16.20.15 apache-pod local-registry:5000/rhel7-ews21:demo minion2/ name=apache Runningd7deb52e-…-52540098677f 172.16.69.239 apache-pod local-registry:5000/rhel7-ews21:demo minion4/ name=apache Running

NAME LABELS SELECTOR IP PORTapache-service name=apache-service name=apache 10.254.220.71 80


Service


Pod

id: apache-servicekind: ServiceapiVersion: v1beta1port: 80containerPort: 80selector: name: apachelabels: name: apache-ser-vice

Service #iptables –L –n –t nat….Chain KUBE-PORTALS-CONTAINER (1 references)target prot opt source destinationREDIRECT tcp -- 0.0.0.0/0 10.254.57.4 /* apache-service */ tcp dpt:80 redir ports 42927REDIRECT tcp -- 0.0.0.0/0 10.254.12.160 /* kubernetes */ tcp dpt:443 redir ports 43044REDIRECT tcp -- 0.0.0.0/0 10.254.34.217 /* kubernetes-ro */ tcp dpt:80 redir ports 49932

Chain KUBE-PORTALS-HOST (1 references)target prot opt source destinationDNAT tcp -- 0.0.0.0/0 10.254.57.4 /* apache-service */ tcp dpt:80 to:192.168.20.12:42927DNAT tcp -- 0.0.0.0/0 10.254.12.160 /* kubernetes */ tcp dpt:443 to:192.168.20.12:43044DNAT tcp -- 0.0.0.0/0 10.254.34.217 /* kubernetes-ro */ tcp dpt:80 to:192.168.20.12:49932

Mininon 2

#iptables –L –n –t natChain KUBE-PORTALS-CONTAINER (1 references)target prot opt source destinationREDIRECT tcp -- 0.0.0.0/0 10.254.57.4 /* apache-service */ tcp dpt:80 redir ports 40144REDIRECT tcp -- 0.0.0.0/0 10.254.12.160 /* kubernetes */ tcp dpt:443 redir ports 42578REDIRECT tcp -- 0.0.0.0/0 10.254.34.217 /* kubernetes-ro */ tcp dpt:80 redir ports 36642

Chain KUBE-PORTALS-HOST (1 references)target prot opt source destinationDNAT tcp -- 0.0.0.0/0 10.254.57.4 /* apache-service */ tcp dpt:80 to:192.168.20.14:40144DNAT tcp -- 0.0.0.0/0 10.254.12.160 /* kubernetes */ tcp dpt:443 to:192.168.20.14:42578DNAT tcp -- 0.0.0.0/0 10.254.34.217 /* kubernetes-ro */ tcp dpt:80 to:192.168.20.14:36642

Mininon 4

Service

min-ion1

min-ion2

min-ion3

min-ion4

Pod : apache

apache80


Pod : apache

apache80


Pod : apache

apache80


Service10.254.220.71:80

Service

min-ion1

min-ion2

min-ion3

min-ion4

Pod : apache

apache80

Container172.16.69.239Pod :

apache

apache80


Pod : apache

apache80


Kube-Proxy Kube-ProxyKube-Proxy Kube-Proxy

docker0

flan-nel.1

12 14

docker0

flan-nel.1

docker0

flan-nel.1

docker0

flan-nel.1

1311

On 172.16.20.15(minion2) request to 10.254.220.71:80

# Curl 10.254.220.71

Kubectl commands• Create

• kubectl create pod/service/rc –f *.yaml/json• Log

• kubectl log pod <pod_id>• Get

• kubectl get pod/service/rc• Describe

• kubectl describe pod <pod_id>• Update

• kubectl update –f *.yaml/json• Delete

• kubectl delete pod/service/rc <id> or –l <label>• Resize replicas

• kubectl resize --current-replicas=3 --replicas=4 rc apache-rc-controller• Rollingupdate

• kubectl rollingupdate apache-rc-controller -f apache-rc-2.yaml --update-period="5s"

Live Demo

•EAP test environment using Docker•Rolling Update

Q&A

•Questions?

Thank you.한국레드햇이주호 과장 [email protected]

mailto:[email protected]