lisa2017 kubernetes: hit the ground running

October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17

Kubernetes:Hit the Ground Running

Chris "mac" McEniry

Administrvia

Goal• Be familiar with the basics of how to use Kubernetes

• Not in scope

• Advanced Usages of Kubernetes

• Administering Kubernetes

Structure• Lecture followed by Watch/Follow

• I'll show it, and you can follow along

• Ask questions as we go along

• We'll take a poll with each section - if 50% are good continuing, we will continue

• Given time and size, unable to do individual attention

• But happy to follow up afterwards

Biases• Focused on Linux based containers

• Expect some (though not much) familiarity with Docker

• Exercises written on MacOS with /bin/sh

• But should work with minor tweaks

Prerequisite Tools• VirtualBox

• docker

• https://store.docker.com/editions/community/docker-ce-desktop-mac

• https://store.docker.com/editions/community/docker-ce-desktop-windows

Basic Kubernetes Tools• kubectl

• curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.5/bin/darwin/amd64/kubectl

• curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.5/bin/windows/amd64/kubectl.exe

• minikube

• https://storage.googleapis.com/minikube/releases/v0.21.0/minikube-darwin-amd64

• https://storage.googleapis.com/minikube/releases/v0.21.0/minikube-windows-amd64.exe

Exercises

https://github.com/cmceniry/lisa2017-kubernetes

Lecture

Containers

Container Info Dump• Lightweight isolation

• More like process than like VM but properties of both

• Same Kernel

• Separate File System (with some mappings)

• Average lifetime of minutes-days

• VMs: Average lifetime of hours-weeks

• Machines: Average lifetime of weeks-years

Container Info Dump• Container : Process with a specific namespace/capability/cgroups configuration

• Namespaces: Create process visibility separation (e.g. what `ps -ef` show, `ls /` looks different)

• Capabilities: Allow/Disallow privileged functions (e.g. raw sockets for packet captures)

• CGroups: Allocate resources for processes (e.g. max cpu, memory, I/O)

• Container Image

• Packaging which becomes the basis for the root file system of a container

• Different ways to maintain it (tarball, layers, etc) depending on runtime which did packaging

Container Info Dump• Multiple Container Runtimes

• Docker : Most common

• Rocket

• CRI-O

• Focus on image management and running individual containers

• Get Image/Start/Stop

Packaging -> 12 Factor Model• Containers can be as simple as a packaging format

• Or can follow https://12factor.net/

• Many usages follow the 12 Factor Model, but some support other apps

Kubernetes Basics

Kubernetes• Container Orchestrator

• Focus on running groups of containers working together

Interface• API Driven (provided by Kubernetes' apiserver)

• Ubiquitous client - comment line `kubectl`

• Have multiple resources: pod, replicaset, deployment, job, volume, etc

• Defined by specification

• Put/Get specifications to/from API server

Desired State• Focus on the *what* (specification passed to the API server) not the *how*

• "I want 5 small compute units doing my web app front end, and 2 large compute units doing my web app database"

• Orchestrator takes definition of "what" and figures out "how" to reconcile it

Controllers• Automation Providers

• Examine the current state of the cluster (via the API server), what the desired state says it should be and takes the necessary steps

• All kinds of controllers:

• Scheduler, Manager

• Cloud Provider(s)

• Ingress

Kubernetes Control Plane Components

Processes• State Storage (etcd) - 1 cluster : Live DB of cluster

• API Server - enough to cover load : How everything interacts

• Scheduler - 3 for redundancy, 1 active : Decides what should be running where

• Controller Manager - 3 for redundancy, 1 active : Spawns internal controllers which handle all heavy lifting functions

• kubelet - 1 per workload node : Runs wherever a workload runs.

• Kube Proxy - 1 per workload node : Handles traffic forwarding into/out of cluster and to known endpoints

• Container Networking - TBD : Handles routing of workloads to each other

Master vs Minion• Master : Core of Control Plane runs here

• Minion : Where the actual workloads run

• kubelet and kube-proxy run on both (depending on control plane installation)

• Lots of variations

• Can have internal/external etcd

• Can run control plane in the cluster even

Add ons• Pluggable cluster features

• network overlay : Provides host<->host connectivity which makes the cluster network look connected

• kube-dns : Provides a DNS domain which provides naming for cluster resources

• das

Kubernetes Resources

Namespaces• Administrative unit

• Hold (most of) the other resources: Pods, Services, CMs, Secrets, etc.

• Apply

Pod• Basic unit of compute

• The "What" of the workload

• Meant for portions of a workload that are tightly coupled

• 1 or more containers scheduled together

• Typically expected to run indefinitely

• Containers sharing a network namespace

• I.e. "host" from a network perspective

Configuration Resources• Under the 12 Factor model, containers end up being immutable, so you need to be able to get

configuration parameters in.

• ConfigMap : key/value collections which can be made available to Containers in Pods

• Secrets : key/value collections which can be made available to Containers in Pods but should be handled carefully

Pod Collections• ReplicaSet : Multiple identically configured pods (differ by IP)

• Deployment : Mutable collection of Pods performing a workload

• Job : Pods meant to run ones

• CronJob : Pods meant to run on a regular but not constant basis

• StatefulSet : Pods meant to have some consistency over their lifetimes (name, Pod IP, etc)

• DaemonSet : Pods meant to running on all nodes (or subset of nodes with a label)

Two Most Common

Service• Abstraction of how to get to a workload

• The "Way" to the workload

• Uses labels to decide membership (which Pods are "behind" the Service)

• For Kubernetes base, this mapping is a L4 load balancer

• But add-ons provide this via DNS and other methods

(Slight aside) Labels• Not a resource, but is metadata on resources

• Every resource can have labels added to it

• Labels: key/value pairs that tag resource

• Used to select subsets of those resources

• "Gimme all Pods with the label `app=web`

• Services say "connect this frontend with all pods with the label `app=web`"

• Metadata also includes name, annotations (structured data), timestamps, status, etc

Other Resources• Authorization: Role, RoleBinding, ClusterRole, ClusterRoleBinding

• StorageClass, PersistentVolume, PersistentVolumeClaim

• Networking: Ingress, NetworkPolicy

• Cluster Management: Node, Cluster (Federation), ComponentStatus (pseudo)

Networking

Unified Network Space• All Pods are reachable from all other Pods from an IP perspective

• Policy may not allow, but common, non-conflicting, and known IP space

• No need to port map ports for Pod<->Pod inside of the cluster

• May need to map Outside->Pod

• Need to map from Container->Pod (container runtime behavior)

• Typically

• Implemented as an overlay but can be done with direct routing

• Must NAT to outside cluster (IPv6 is coming not full support yet)

IPs• Cluster IP Space : Used to provide Unified Network Space. Pool of Pod IPs

• Service IP Space : Used to provide area to map services into

• Node IPs : The IPs assigned to master/minions

• Typically used when referring to getting into/out of the cluster

IPTables• Any mapping in the cluster is done via IPTables

• Handles Service IP to backend Pod IPs

• Handles inside->outside IP NAT

• Handles outside->inside IP NAT

• Controlled by kube-proxy (not really a proxy anymore)

Useful Pointers

Common things to remember• Pods : multiple containers working closely together

• Pod IP : Ultimately how traffic gets into a workload. One CIDR over a cluster.

• IPTables : How cube proxy maps everything into/out of the cluster (and how it maps services)

• Controllers : Entities that do a piece of automation

• Labels and Selectors : Ways to classify resources

• IPTables, IPTables, IPTables

Exercises

EX00: Starting minikube• Purpose

• Make sure everything is up and running

EX00: Starting minikube$ minikube startStarting local Kubernetes v1.7.0 cluster...Starting VM...Downloading Minikube ISO 97.80 MB / 97.80 MB [=====================================] 100.00% 0sGetting VM IP address...Moving files into cluster...Setting up certs...Starting cluster components...Connecting to cluster...Setting up kubeconfig...Kubectl is now configured to use the cluster.

EX00: Starting minikube$ minikube statusminikube: Runninglocalkube: Runningkubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100$$ kubectl versionClient Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-07-26T00:12:31Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Is everything running ok?

EX00: Starting minikube$ kubectl get csNAME STATUS MESSAGE ERRORscheduler Healthy okcontroller-manager Healthy oketcd-0 Healthy {"health": "true"}$$$ kubectl cluster-infoKubernetes master is running at https://192.168.99.100:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

EX01: helloshell• Purpose

• Show how to run a simple command

• What's going on when you run the command

• Demonstrate pod, replicaset, deployment

EX01: helloshell$ kubectl run -it --image=busybox bb1 /bin/shIf you don't see a command prompt, try pressing enter./ # psPID USER TIME COMMAND 1 root 0:00 /bin/sh 7 root 0:00 ps/ #

Is everything running ok?

EX01: helloshell$ kubectl get podNAME READY STATUS RESTARTS AGEbb1-1176220718-z09mj 1/1 Running 1 46s$$$$$ kubectl get rsNAME DESIRED CURRENT READY AGEbb1-1176220718 1 1 1 19s$$$ kubectl get deployNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEbb1 1 1 1 1 1m

What's this actually running?

*-##########-XXXXX format tends to look like a deployment/replicaset

Why 1?

EX01: helloshell$ kubectl attach bb1-1176220718-z09mj -c bb1 -i -tIf you don't see a command prompt, try pressing enter./ # Session ended, resume using 'kubectl attach bb1-1176220718-z09mj -c bb1 -i -t' command when the pod is running$$$$ kubectl attach bb1-1176220718-z09mj -c bb1 -i -tIf you don't see a command prompt, try pressing enter. error: unable to upgrade connection: container bb1 not found in pod bb1-1176220718-z09mj_default$ $ kubectl get pod bb1-1176220718-z09mjNAME READY STATUS RESTARTS AGEbb1-1176220718-z09mj 0/1 Completed 2 10m

Let's try exiting and entering quickly

What's this error?

What does it mean when the pod is running?

0/1 means it's in the middle of (re-)starting

EX01: helloshell$ kubectl get pod bb1-1176220718-z09mj -o yamlapiVersion: v1kind: Pod...spec: containers: ... restartPolicy: Always

Pod description says it's going to try to restart

EX01: helloshell$ kubectl delete deploy/bb1deployment "bb1" deleted

Let's clean up some

EX02: Official Introduction• Purpose

• Connect docker and kubernetes

• Build artifacts that can go into kubernetes

• Reinforce pod, replicaset, deployment

• Demonstrate services

• From: https://kubernetes.io/docs/tutorials/stateless-application/hello-minikube/

EX02: Official Introduction $ eval $(minikube docker-env)$ docker psCONTAINER ID IMAGE COMMAND6d5fac7595bd gcr.io/google_containers/pause-amd64:3.0 "/pause"...

Connect to minikube's docker daemon

Now we can interact with it just as if it was a local docker daemon

EX02: Official Introduction$ cd ex02$ lsDockerfile server.js$$ docker build -t intro:0.0.1 .Sending build context to Docker daemon 3.072kBStep 1 : FROM node:6.9.26.9.2: Pulling from library/node75a822cd7888: Pull complete57de64c72267: Pull complete...Step 4 : CMD node server.js ---> Running in 22b57d427b1c ---> 280abd363febRemoving intermediate container 22b57d427b1cSuccessfully built 280abd363feb

Build a container image from the official example

EX02: Official Introduction$ kubectl run intro --image=intro:0.0.1 --port=8080$$ kubectl get deploymentsNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEintro 1 1 1 1 7s$$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1197849725-75kk9 1/1 Running 0 10s$

Start the image

So, it's running - now what?

EX02: Official Introduction$ kubectl expose deploy/intro --type=NodePortservice "intro" exposed$$$ kubectl get svcNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGEintro 10.0.0.238 <nodes> 8080:30561/TCP 6skubernetes 10.0.0.1 <none> 443/TCP 9h$$$ minikube service introOpening kubernetes service default/intro in default browser...$$$ minikube service intro --urlhttp://192.168.99.100:30561

Make it available over the network

Tells to map it to a port on all (one) of the nodes

Automatically opens browser

Or get the URL yourself

We're going to leave this running, for the next exercise...

EX03: The Dashboard Add-on• Purpose

• Demonstrate the dashboard add-on

• Demonstrate minikube dashboard shortcuts

EX03: The Dashboard Add-on$ minikube dashboardOpening kubernetes dashboard in default browser...$$$ minikube dashboard --urlhttp://192.168.99.100:30000$

Start up the dashboard

Automatically opens browser

As before, can get it yourself

But before, we did `minikube service $NAME`...

EX03: The Dashboard Add-on$ kubectl get service -n kube-systemNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGEkube-dns 10.0.0.10 <none> 53/UDP,53/TCP 9hkubernetes-dashboard 10.0.0.114 <nodes> 80:30000/TCP 9h$$$ minikube service -n kube-system kubernetes-dashboardOpening kubernetes service kube-system/kubernetes-dashboard in default browser...$ minikube service -n kube-system kubernetes-dashboard --urlhttp://192.168.99.100:30000

There is a `kubernetes-dashboard` service running; it's just in the kibe-system namespace (we'll come back to that...)

`minikube dashboard` is a shortcut for `minikube service...` for the dashboard

EX03: The Dashboard Add-on

EX04: Add-ons• Purpose

• Explore the add-ons and add-ons manager

• Explore the kibe-system namespace

EX04: Add-ons$ kubectl get -n kube-system podsNAME READY STATUS RESTARTS AGEkube-addon-manager-minikube 1/1 Running 0 21hkube-dns-910330662-rnwgp 3/3 Running 0 21hkubernetes-dashboard-tlh94 1/1 Running 0 21h$$$ minikube addons list- ingress: disabled- dashboard: enabled- heapster: disabled- kube-dns: enabled- registry: disabled- registry-creds: disabled- addon-manager: enabled- default-storageclass: enabled

Already saw kibe-system services - what about pods?

This are background processes which are managed by the qinikube add-ons manager

EX04: Add-ons$ minikube addons enable heapsterheapster was successfully enabled$$$ kubectl -n kube-system get podsNAME READY STATUS RESTARTS AGEheapster-t00zx 1/1 Running 0 2sinfluxdb-grafana-ll71w 2/2 Running 0 2skube-addon-manager-minikube 1/1 Running 0 22hkube-dns-910330662-rnwgp 3/3 Running 0 22hkubernetes-dashboard-tlh94 1/1 Running 0 22h

Let's turn on something else

Check on the pods again

EX04: Add-ons• addon-manager : Controller which provides these add-ons

• dashboard : Web interface for cluster information and status

• kube-dns : Provides cluster DNS mapping (we'll come back to this)

• heapster : Gathers container and node statistics

• registry : Can run a container image registry

• default-storageclass : Provides a simply host path persistent volume

• ingress : Provides a Layer 7 load balancer as Kubernetes primitive

• registry-creds : Simplified way to provide container registry user/password for image pulls

EX04: Add-ons

EX05: Working with pods• Purpose

• Explore multiple ways of seeing pod information

• Explore the pod spec

EX05: Working with pods$ kubectl get podNAME READY STATUS RESTARTS AGEintro-1197849725-g22tx 1/1 Running 0 15m$$$ kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODEintro-1197849725-g22tx 1/1 Running 0 15m 172.17.0.4 minikube$$$ kubectl get pods -o namepods/intro-1197849725-g22tx$

Get pods - normal

Get pods - show Pod IP and Node

Get pods - show just name (good for shell loops)

EX05: Working with pods$ kubectl describe pod/intro-1197849725-g22txName: intro-1197849725-g22txNamespace: defaultNode: minikube/192.168.99.100Start Time: Sat, 09 Sep 2017 13:26:55 -0700Labels: pod-template-hash=3978227742

run=introAnnotations: ...Status: RunningIP: 172.17.0.10Created By: ReplicaSet/intro-1197849725Controlled By: ReplicaSet/intro-1197849725Containers: intro: Container ID: docker://... Image: intro:0.0.2 Image ID: docker://sha256:... Port: 8080/TCP State: Running...

This is `describe pod`. It gives you some human readable information about the pod.

EX05: Working with pods$ kubectl get pods/intro-1197849725-g22tx -o yamlapiVersion: v1kind: Podmetadata: name: intro-1197849725-g22tx namespace: default ...spec: containers: - image: intro:0.0.1 ...status: hostIP: 192.168.99.100 podIP: 172.17.0.4 ...

This is what a pod spec looks like. This can be used for specific search/display or to configure the system.

EX05: Working with pods$ kubectl get pod -o=custom-columns=NAME:.metadata.name,IP:.status.podIPNAME IPintro-1197849725-g22tx 172.17.0.4

Show just the name and podIP

EX05: Working with pods $ cat redis.yamlapiVersion: v1kind: Podmetadata: name: redis-manualspec: containers: - image: redis:4.0.1 name: redis$$ kubectl apply -f redis.yamlpod "redis-manual" created$ kubectl get podsNAME READY STATUS RESTARTS AGEredis-manual 1/1 Running 0 2s...

Write a spec file manually

Apply it to the cluster

EX05: Working with pods $ kubectl apply -f redis.yamlpod "redis-manual" created$$$$$ kubectl delete -f redis.yamlpod "redis-manual" deleted$$$$$$ kubectl apply -f redis.yamlpod "redis-manual" created$

$ kubectl get pods -w | grep redisredis-manual 0/1 Pending 0 0sredis-manual 0/1 Pending 0 0sredis-manual 0/1 ContainerCreating 0 0sredis-manual 1/1 Running 0 1s

redis-manual 1/1 Terminating 0 7sredis-manual 0/1 Terminating 0 8sredis-manual 0/1 Terminating 0 9sredis-manual 0/1 Terminating 0 9s

redis-manual 0/1 Pending 0 15sredis-manual 0/1 Pending 0 15sredis-manual 0/1 ContainerCreating 0 15sredis-manual 1/1 Running 0 16s

We can also -w(atch) the pod changes

EX05: Working with pods $ kubectl get events | grep redis-manual3m 3m 1 redis-manual Pod Normal Scheduled default-scheduler Successfully assigned redis-manual to minikube

3m 3m 1 redis-manual Pod Normal SuccessfulMountVolume kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-3j744"

3m 3m 1 redis-manual Pod spec.containers{redis} Normal Pulled kubelet, minikube Container image "redis:4.0.1" already present on machine

3m 3m 1 redis-manual Pod spec.containers{redis} Normal Created kubelet, minikube Created container

3m 3m 1 redis-manual Pod spec.containers{redis} Normal Started kubelet, minikube Started container

3m 3m 1 redis-manual Pod spec.containers{redis} Normal Killing kubelet, minikube Killing container with id docker://redis:Need to kill Pod

Can see the same a -w(atch) and more in the events

EX06: Working in a container• Purpose

• Explore starting points for debugging

• Explore how to get logs

• Explore how to get inside a container

EX06: Working in a container$ kubectl logs redis-manual...:32.344 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo...:32.344 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=1, just started...:32.344 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf...:32.345 * Running mode=standalone, port=6379....:32.345 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128....:32.345 # Server initialized...:32.345 * Ready to accept connections

Getting "logs" == stdout and stderr

Convention of 12 Factor logging approach. https://12factor.net/logs

EX06: Working in a container$ kubectl exec -it redis-manual /usr/local/bin/redis-cli127.0.0.1:6379> set foo 10OK127.0.0.1:6379> get foo"10"127.0.0.1:6379>$$$ kubectl exec -it redis-manual /usr/local/bin/redis-cli127.0.0.1:6379> get foo"10"

How to work on the redis container?

It persists across invocations of the client command

EX06: Working in a container$ kubectl delete pod/redis-manualpod "redis-manual" deleted$ kubectl apply -f ./redis.yamlpod "redis-manual" created$ kubectl exec -it redis-manual /usr/local/bin/redis-cli127.0.0.1:6379> get foo(nil)

But does not persist across invocations of the pod itself

EX07: Deployment replicas• Purpose:

• Explore deployment keeping replicas running

• Explore adding/removing replicas from a deployment

EX07: Deployment replicas$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1197849725-g22tx 1/1 Running 0 30mredis-manual 1/1 Running 0 10m$$$ kubectl get pods -o name | xargs kubectl deletepod "intro-1197849725-g22tx" deletedpod "redis-manual" deleted$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1197849725-g22tx 1/1 Terminating 0 30mintro-1197849725-v78kg 1/1 Running 0 17s

Let's do a little cleanup

A new intro pod is already around

EX07: Deployment replicas$ kubectl get deploy introNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEintro 1 1 1 1 30m$$$$ kubectl get podNAME READY STATUS RESTARTS AGEintro-1197849725-v78kg 1/1 Terminating 0 2m$

Deployment tries to keep CURRENT equal to DESIRED

Delete the deployment to make the pod go away

EX07: Deployment replicas$ kubectl run intro --image=intro:0.0.1 --port=8080 --replicas=3deployment "intro" created$ kubectl get deployNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEintro 3 3 3 3 13s$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1197849725-bl6qs 1/1 Running 0 17sintro-1197849725-gdr6f 1/1 Running 0 17sintro-1197849725-qm7zd 1/1 Running 0 17s

Let's start it with more instances

replicas == pod count

EX07: Deployment replicas$ kubectl delete po/intro-1197849725-qm7zdpod "intro-1197849725-qm7zd" deleted$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1197849725-bl6qs 1/1 Running 0 1mintro-1197849725-gdr6f 1/1 Running 0 1mintro-1197849725-l50k3 1/1 Running 0 3sintro-1197849725-qm7zd 1/1 Terminating 0 1m$ kubectl get deploy introNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEintro 3 3 3 3 1m

Delete a pod again

The deployment will do what it needs to to get the count back to 3

EX07: Deployment replicas$ kubectl scale deploy/intro --replicas=1deployment "intro" scaled$ kubectl get deploy/introNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEintro 1 1 1 1 2m$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1197849725-bl6qs 1/1 Terminating 0 3mintro-1197849725-gdr6f 1/1 Running 0 3mintro-1197849725-l50k3 1/1 Terminating 0 1m$$

Scale it down to 1

EX08: Deployment updates• Purpose

• Show how to update a deployment

EX08: Deployment updates

Let's update Hello World to be a bit more specific.

EX08: Deployment updates$ grep Hello server.js response.end('Hello Kubernetes Tutorial!');$ docker build -t intro:0.0.2 .Sending build context to Docker daemon 3.072kBStep 1 : FROM node:6.9.2 ---> faaadb4aaf9bStep 2 : EXPOSE 8080 ---> Using cache ---> 20e3088f6122Step 3 : COPY server.js . ---> 9aa0164faa7eRemoving intermediate container abe7416c8707Step 4 : CMD node server.js ---> Running in 4d672a4e6fac ---> 1df0203ce037Removing intermediate container 4d672a4e6facSuccessfully built 1df0203ce037

First, we need a new image to update to.

EX08: Deployment updates$ kubectl scale deploy/intro --replicas=3deployment "intro" scaled$$ # kubectl get pods -w ### watch the deployment as it happens$$$ kubectl set image deploy/intro intro=intro:0.0.2deployment "intro" image updated$ kubectl rollout status deploy/introdeployment "intro" successfully rolled out$$ minikube service intro

Next, let's make sure we have some additional copies for resilience

EX08: Deployment updatesintro-3978227742-hllw8 0/1 Pending 0 0sintro-1197849725-j3l1c 1/1 Terminating 0 1mintro-3978227742-hllw8 0/1 Pending 0 0sintro-3978227742-hllw8 0/1 ContainerCreating 0 0sintro-3978227742-6kb68 0/1 Pending 0 0sintro-3978227742-6kb68 0/1 Pending 0 0sintro-3978227742-6kb68 0/1 ContainerCreating 0 0sintro-3978227742-hllw8 1/1 Running 0 0sintro-1197849725-lt7fs 1/1 Terminating 0 1mintro-3978227742-5d5w9 0/1 Pending 0 0sintro-3978227742-5d5w9 0/1 Pending 0 0sintro-3978227742-5d5w9 0/1 ContainerCreating 0 0sintro-3978227742-5d5w9 1/1 Running 0 1sintro-1197849725-2nzz4 1/1 Terminating 0 2mintro-3978227742-6kb68 1/1 Running 0 1s

Our deployment strategy (default rollingUpdate) will create new Pods before deleting the old ones, and it will roll over some of the pods "slowly". (In this exercise, the pods come up too quickly so not much waiting.)

EX09 Pod information inside• Purpose

• Show how to expose information to pod

• Explore the deployment specification

• Explore the `edit` command

EX09 Pod information inside

Since I have 3 pods, how do I know which one I'm hitting? Let's add the pod IP to our response.

EX09 Pod information inside$ grep Hello server.js response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!\n');$ docker build -t intro:0.0.3 .Sending build context to Docker daemon 3.072kBStep 1 : FROM node:6.9.2 ---> faaadb4aaf9bStep 2 : EXPOSE 8080 ---> Using cache ---> 20e3088f6122Step 3 : COPY server.js . ---> 83e6090ec153Removing intermediate container 76bf52dc48dcStep 4 : CMD node server.js ---> Running in c08880cc596d ---> e2c588c47a0aRemoving intermediate container c08880cc596dSuccessfully built e2c588c47a0a$$ kubectl set image deploy/intro intro=intro:0.0.3deployment "intro" image updated

Start by adding a new image (0.0.3) which pulls an environment variable called PODIP

And roll this out

It's updated, but we haven't defined the environment variable in it yet.

EX09 Pod information inside$ kubectl get deploy/intro -o yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata: name: intro ...spec: replicas: 3 strategy: rollingUpdate: ... template: spec: containers: - image: intro:0.0.3 name: intro ports: - containerPort: 8080 protocol: TCP ...

Deployments have specs just like pods do

The pod spec is nested inside of the deployment spec

EX09 Pod information inside$ kubectl edit deploy/intro

spec:... template: spec: containers: - image: intro:0.0.3 name: intro env: - name: PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP...

Opens up the EDITOR

Once it's written and EDITOR is exited, it'll save and cycle the pods

EX09 Pod information inside...deployment "intro" edited$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-1725170555-306cj 1/1 Terminating 0 28mintro-1725170555-pw8zd 1/1 Terminating 0 28mintro-1725170555-qbctx 1/1 Terminating 0 28mintro-3265745252-07l0b 1/1 Running 0 3sintro-3265745252-0qw03 1/1 Running 0 4sintro-3265745252-sxz2s 1/1 Running 0 4s$$ minikube service intro...

The pods cycle...

And we are now showing the Pod IP

EX09 Pod information inside$ kubectl get pod -o=custom-columns=NAME:.metadata.name,IP:.status.podIPNAME IPintro-3265745252-07l0b 172.17.0.11intro-3265745252-0qw03 172.17.0.9intro-3265745252-sxz2s 172.17.0.10

And confirm the Pod IPs

EX09 Pod information inside$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.11!$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.9!$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.10!$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.9!$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.10!$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.11!

Run it from the command line since browsers will pool the connection...

EX10 Configmaps• Purpose

• Explore configmaps

• Explore `kubectl apply`

EX10 Configmaps$ grep -A 2 Hello server.js response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!\n' + 'The configuration variable is ' + process.env.CONFIGVAR + '\n' );$ docker build -t intro:0.0.4 .Sending build context to Docker daemon 3.072kBStep 1 : FROM node:6.9.2 ---> faaadb4aaf9bStep 2 : EXPOSE 8080 ---> Using cache ---> 20e3088f6122Step 3 : COPY server.js . ---> 5f8dad93c9b3Removing intermediate container 29b6ad3411b5Step 4 : CMD node server.js ---> Running in c6911d08376d ---> f72a2166a111Removing intermediate container c6911d08376dSuccessfully built f72a2166a111

Update our server to output something with more environment variables in it

EX10 Configmaps$ kubectl set image deploy/intro intro=intro:0.0.4deployment "intro" image updated$$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-3265745252-0qw03 1/1 Terminating 0 24mintro-3265745252-7wg78 1/1 Terminating 0 29sintro-3265745252-sxz2s 1/1 Terminating 0 24mintro-4010436465-3sbgd 1/1 Running 0 4sintro-4010436465-lr2qh 1/1 Running 0 2sintro-4010436465-nchsj 1/1 Running 0 4s$$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.8!The configuration variable is undefined

See that it's using it, but like with the Pod IP, it's not being set yet

EX10 Configmaps$ kubectl create configmap --from-literal=configvar=valuea introconfigmap "intro" created$$ kubectl edit deploy/intro...spec: template: spec: containers: - env: - name: CONFIGVAR valueFrom: configMapKeyRef: name: intro key: configvar...deployment "intro" edited

Create a configmap with one key/value in it

Map that configmap's key in as an environment variable

EX10 Configmaps$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-2942694475-948fj 1/1 Running 0 6sintro-2942694475-9tf89 1/1 Running 0 6sintro-2942694475-vh007 1/1 Running 0 6sintro-3265745252-07l0b 1/1 Terminating 0 3sintro-3265745252-0qw03 1/1 Terminating 0 4sintro-3265745252-sxz2s 1/1 Terminating 0 4s$$$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.10!The configuration variable is valuea

Deployment changed, so pods roll

And now it's using the CONFIGVAR environment variable to output value

EX10 Configmaps$ kubectl get cm/intro -o yamlapiVersion: v1data: configvar: valueakind: ConfigMapmetadata: creationTimestamp: 2017-09-10T05:36:09Z name: intro namespace: default resourceVersion: "170767" selfLink: /api/v1/namespaces/default/configmaps/intro uid: f2f0527d-95e9-11e7-b635-080027358e48$ kubectl get cm/intro -o yaml > intro-cm.yaml

Another way is to use the configmap spec like pod and deployment. Can get that by looking at what's in there already.

Save that out to a file

EX10 Configmaps$ vi intro.yaml...apiVersion: v1data: configvar: valueskind: ConfigMapmetadata: name: intro...$ $ kubectl delete cm/infoconfigmap "intro" deleted$$ kubectl apply -f intro-cm.yamlconfigmap "intro" created

Reduce it down to take out the Kuberentes server decoration

Delete the old info

`apply` tries to create/update the resource in sync with the file

In this case, it creates

EX10 Configmaps$ vi intro.yaml...apiVersion: v1data: configvar: valuebkind: ConfigMapmetadata: name: intro...$$ kubectl apply -f intro-cm.yamlconfigmap "intro" created$ $ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.11!The configuration variable is valuea

Let's update the configvar

And testing... we see that it hasn't updated

EX10 Configmaps$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-2942694475-t0dk8 1/1 Running 0 9mintro-2942694475-tlpvm 1/1 Running 0 9mintro-2942694475-vfwwf 1/1 Running 0 9m$$$ kubectl get pods -o name | xargs kubectl deletepod "intro-2942694475-t0dk8" deletedpod "intro-2942694475-tlpvm" deletedpod "intro-2942694475-vfwwf" deleted$$$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.10!The configuration variable is valuec

A configmap change isn't readily identified as causing the deployment to change, these pods don't get restarted automatically.

Delete the pods manually

Test again, and we see that it has changed

EX10 Configmaps• Can also define config maps from files

• Include the whole file verbatim: kubectl create cm test --from-file=configs=/path/to/file

• Include the file as a list of key/value pairs: kubectl create cm test --from-env-file=/path/to/file

EX11: Secrets• Purpose

• Explore the Secrets resource

EX11: Secrets$ kubectl create secret generic intro --from-literal=password=reallysecretsecret "intro" created$$ kubectl get secret intro -o yamlapiVersion: v1data: password: cmVhbGx5c2VjcmV0kind: Secretmetadata: name: intro namespace: default ...type: Opaque

Secert is very similar to the configmap, but it's meant to have some meaning behind it (and handling is in progress)

Stores as base64 encoded values available from the API

EX11: Secrets$ grep -A 2 Hello server.js response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!\n' + 'The secret password is "' + process.env.PW + '"\n' );$ docker build -t intro:0.0.5 .Sending build context to Docker daemon 3.072kBStep 1 : FROM node:6.9.2 ---> faaadb4aaf9bStep 2 : EXPOSE 8080 ---> Using cache ---> 20e3088f6122Step 3 : COPY server.js . ---> cb2fb7acc119Removing intermediate container 409c93df3ec7Step 4 : CMD node server.js ---> Running in 35465f243ef9 ---> 566294badefdRemoving intermediate container 35465f243ef9Successfully built 566294badefd

Can use it the same way - set up the secret as an environment variable

EX11: Secrets$ cat intro-deploy.yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata:...spec: ... template: .. spec: containers: - env: ... - name: PW valueFrom: secretKeyRef: name: intro key: password image: intro:0.0.5

Updated intro deployment specification

Make the password available to the app as part of the environment

Update to our latest build

EX11: Secrets$ kubectl apply -f intro-deploy.yamlWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl applydeployment "intro" configured$$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-2942694475-fpnfh 1/1 Terminating 0 30mintro-2942694475-gqdvf 1/1 Terminating 0 30mintro-2942694475-rxn92 1/1 Terminating 0 30mintro-3353884051-kdz2c 1/1 Running 0 2mintro-3353884051-nnkln 1/1 Running 0 2mintro-3353884051-rln6s 1/1 Running 0 2m$$ minikube service introOpening kubernetes service default/intro in default browser...

Update the new deploy spec with `apply`

Updated deployment causes pods to roll

And see if it worked...

EX11: Secrets

It worked!

EX11: Secrets• You can change access to secrets separate from access to configmaps (see RBAC)

• Exposing via the environment may leak it (env available in other ways) --- we'll look at that next

• There is work to protect the secrets more

• Not allow any node access to the secret -- only ones where the secret is scheduled

• Sealing it all the way to the process

• Can use external secret stores - Vault, CyberArk, KMS, but mileage may vary

EX12: Volumes• Purpose

• Explore the volumes, volumeMounts fields in the spec

• Explore secrets, configmaps as mounts

EX12a: Volumes$ cat server.jsvar http = require('http');

var fs = require('fs');var password = fs.readFileSync('/data/password', 'UTF8');

var handleRequest = function(request, response) { console.log('Received request for URL: ' + request.url); response.writeHead(200); response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!\n' + 'The secret password from env is "' + process.env.PW + '"\n' + 'The secret password from fs is "' + password +'"\n' );};var www = http.createServer(handleRequest);www.listen(8080);

Update to read a secret from a file system path `/data/password`

EX12a: Volumes$ cat intro-deploy.yamlkind: Deployment...spec: template: ... spec: volumes: - name: intro secret: secretName: intro containers: - volumeMounts: - name: intro readOnly: true mountPath: /data ... image: intro:0.0.6

Update the deployment spec to map the intro secret to `/data`. This puts the `password` key at `/data/password`

EX12a: Volumes$ docker build -t intro:0.0.6 .Sending build context to Docker daemon 5.632kB...$ kubectl apply -f intro-deploy.yamldeployment "intro" configureddeployment "intro" configured$ kubectl get podsNAME READY STATUS RESTARTS AGEintro-2431729973-nn0t0 1/1 Running 0 4s...intro-538233644-8tj16 1/1 Terminating 0 4m...$ minikube service intro

Deploy it all - build image, apply deployment, check for changed pods, and open the browser

EX12a: Volumes

EX12b: Volumes$ cat server.jsvar http = require('http');

var fs = require('fs');var configvar = fs.readFileSync('/cm/configvar', 'UTF8');

var handleRequest = function(request, response) { console.log('Received request for URL: ' + request.url); response.writeHead(200); response.end('Hello Kubernetes Tutorial from ' + process.env.PODIP + '!\n' + 'The configvar from fs is "' + configvar + '"\n' );};var www = http.createServer(handleRequest);www.listen(8080);

Same can be done for config map

EX12b: Volumes$ cat intro-deploy.yamlkind: Deployment....spec: ... template: ... spec: volumes: ... - name: cm configMap: name: intro containers: - volumeMounts: ... - name: cm readOnly: true mountPath: /cm

Define config map volume in deployment spec

EX12b: Volumes$ docker build -t intro:0.0.7 .Sending build context to Docker daemon 5.12kB...$ kubectl apply -f intro-deploy.yamldeployment "intro" configured$ kubectl get podsNAME READY STATUS RESTARTS AGE...intro-1784550256-3qvbn 0/1 ContainerCreating 0 3s...intro-2431729973-m7fxw 1/1 Terminating 0 3m...

Redeploy

EX12b: Volumes$ curl http://192.168.99.100:30561/Hello Kubernetes Tutorial from 172.17.0.3!The configvar from fs is "valuec"

And test...

EX12: Volumes• Additional Volume types, but depend on environment

• HostPath volume

• Local volume

• NFS and NAS volumes

• Ceph, Gluster, ScaleIO, etc volumes

• Cloud volumes - AWS EBS/EFS, GCP Persistent Disk, Azure Disk/File

EX13: Stateful Sets• Purpose

• Explore support for applications expecting consistent IPs

EX13: Stateful Sets$ cat redis-statefulset.yamlapiVersion: apps/v1beta1kind: StatefulSetmetadata: name: redisspec: serviceName: redis replicas: 1 template: metadata: labels: app: redis spec: containers: - name: redis image: redis:4.0.1 volumeMounts: - mountPath: /data name: redis-data volumes: - name: redis-data hostPath: path: /data

StatefulSet spec is similar to Deployment where it has a nested Pod spec inside of it

hostPath volume creates a place to preserve data (separate from the Name/IP preservation)

EX13: Stateful Sets$ kubectl apply -f redis-statefulset.yamlstatefulset "redis" created$ kubectl get statefulsetNAME DESIRED CURRENT AGEredis 1 1 24s$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODEintro-1784550256-300fz 1/1 Running 0 1m 172.17.0.10 minikubeintro-1784550256-cxq4v 1/1 Running 0 1m 172.17.0.9 minikubeintro-1784550256-lc4l8 1/1 Running 0 1m 172.17.0.11 minikuberedis-0 1/1 Running 0 1m 172.17.0.3 minikube

Apply just like any of the others

Pod IP is allocatedCreate Pod names based on statefulset name with an identifier after

EX13: Stateful Sets$ kubectl run -it --image=redis:4.0.1 shell /bin/shIf you don't see a command prompt, try pressing enter.## redis-cli -h 172.17.0.3172.17.0.3:6379> 172.17.0.3:6379> set foo barOK172.17.0.3:6379> get foo"bar"172.17.0.3:6379> saveOK172.17.0.3:6379>#Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell -i -t' command when the pod is running

Let's operate inside of the pod a little bit

Connect to the server pod based on IP

Set some data and to check back later

Make sure the data is saved to disk

EX13: Stateful Sets$ kubectl delete pod redis-0pod "redis-0" deleted$ kubectl get pods/redis-0 -o wide -wNAME READY STATUS RESTARTS AGE IP NODEredis-0 1/1 Terminating 0 13s 172.17.0.3 minikuberedis-0 0/1 Terminating 0 14s <none> minikuberedis-0 0/1 Terminating 0 23s <none> minikuberedis-0 0/1 Terminating 0 23s <none> minikuberedis-0 0/1 Pending 0 4s <none> <none>redis-0 0/1 Pending 0 4s <none> minikuberedis-0 0/1 ContainerCreating 0 4s <none> minikuberedis-0 1/1 Running 0 5s 172.17.0.3 minikube

Delete the pod, and it's recreated automatically with the same name/IP

EX13: Stateful Sets$ kubectl attach shell-2621852270-816gf -c shell -itIf you don't see a command prompt, try pressing enter.# redis-cli -h 172.17.0.3172.17.0.3:6379> get foo"bar"172.17.0.3:6379>#

Check back in the new pod and see if the connection IP and data is preserved

EX13: Stateful Sets• Tied together with volumes and storage classes, StatefulSets can help with non-12 Factor Apps

• Downsides

• Can't pick IP ahead of time

• Affects pod scheduling (has to map to existing node)

EX14: Services• Purpose

• Explore the Service spec

• Explore cluster DNS

EX14: Services$ kubectl apply -f redis.yamlpod "redis" created$$ kubectl get pods/redis -o wideNAME READY STATUS RESTARTS AGE IP NODEredis 1/1 Running 0 21s 172.17.0.3 minikube$ kubectl attach shell-2621852270-816gf -c shell -itIf you don't see a command prompt, try pressing enter.# redis-cli -h 172.17.0.3172.17.0.3:6379> GET foo"bar"172.17.0.3:6379>#Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell -i -t' command when the pod is running

EX14: Services$ kubectl delete pod/redispod "redis" deleted$ $ kubectl scale deploy/intro --replicas=5deployment "intro" scaled$ $ kubectl apply -f redis.yamlpod "redis" created$ kubectl get pod/redis -o wideNAME READY STATUS RESTARTS AGE IP NODEredis 1/1 Running 0 9s 172.17.0.12 minikube$

New pod has new IP

Delete to free up the IP

Make something else take up the existing IP (.3)

Recreate

EX14: Services$ kubectl attach shell-2621852270-816gf -c shell -itIf you don't see a command prompt, try pressing enter.## redis-cli -h 172.17.0.3Could not connect to Redis at 172.17.0.3:6379: Connection refusedCould not connect to Redis at 172.17.0.3:6379: Connection refusednot connected> exit## redis-cli -h 172.17.0.12172.17.0.12:6379> GET foo"bar"172.17.0.12:6379>#Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell -i -t' command when the pod is running

Let's try to get the data again

Using the old IP address will fail

Try the new iP

Data is still there

EX14: Services$ kubectl get serviceNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGEintro 10.0.0.238 <nodes> 8080:30561/TCP 1hkubernetes 10.0.0.1 <none> 443/TCP 1h$ kubectl get service/intro -o yamlapiVersion: v1kind: Servicemetadata: labels: run: intro name: intro ...spec: clusterIP: 10.0.0.238 ports: - nodePort: 30561 port: 8080 protocol: TCP targetPort: 8080 selector: run: intro type: NodePort ...

Look at the existing services

This came from the `expose` in EX02

EX14: Services$ kubectl expose pod redis --port 6379service "redis" exposed$ kubectl get service/redisNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGEredis 10.0.0.184 <none> 6379/TCP 6s$ kubectl get service/redis -o yamlapiVersion: v1kind: Servicemetadata: labels: app: redis name: redis ...spec: clusterIP: 10.0.0.184 - port: 6379 protocol: TCP targetPort: 6379 selector: app: redis ...

This is the IP to use to connect to

EX14: Services$ kubectl get service/redis -o yamlapiVersion: v1kind: Servicespec: selector: app: redis ...$ kubectl get pods/redis -o yamlapiVersion: v1kind: Podmetadata: labels: app: redis name: redis ...

How the service maps to the pods

EX14: Services

• Ensure that the `kube-dns` add-on is running

• This provides a mapping from $SERVICENAME to $IP

• FQDN: $SERVICE_NAME.$NAMESPACE.svc.$CLUSTER_DOMAIN

• So can use DNS instead of L4 mappings

$ minikube addons list | grep kube-dns- kube-dns: enabled

EX14: Services$ kubectl attach shell-2621852270-816gf -c shell -itIf you don't see a command prompt, try pressing enter.# # redis-cli -h 10.0.0.18410.0.0.184:6379> GET foo"bar"10.0.0.184:6379>## redis-cli -h redisredis:6379> GET foo"bar"redis:6379>#Session ended, resume using 'kubectl attach shell-2621852270-816gf -c shell -i -t' command when the pod is running

Try to access it via the IP address of the service

Try to access it via the DNS name

Where to go from here?

Where to go from here?• Topics

• Running the Kubernetes cluster itself

• Persistent Volumes

• Ingresses

• Access Control

• Operators

• Helm

• Multicontainer Pods, Sidecars

October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17

Remember to fill inyour tutorial evaluations!

Thank You!

F2 - Kubernetes : Hit the Ground RunningChris "mac" McEniry

lisa2017 kubernetes: hit the ground running

Technology

kubevirt - back to vms · virtual network function...

hit the ground running: financial planning after deployment

hit the ground running in 2013

instaclustr cassandra and kubernetes presention...• k8s...

running k8s on raspberry pi’s · 2020-07-08 · summary...

running sap data hub on kubernetes with suse caas platform

hit the ground running with interbase

running cloud native applications on digitalocean...

kubernetes: up and running - tectonic by coreos · this...

kubernetes: up and running...introduction kubernetes is an...

ion ond kubernetes - amazon web services · this work is...

hit the ground running with snmp - usenix.org

running production-grade kubernetes on aws

running mysql in kubernetes %2f openshift · running mysql...

kubernetes: up and running - aaron | msft · kubernetes: up...

hit the ground running 2013 - strategic thinking

russian prep: «hit the ground running!»

running legacy vm’s along with containers in...

hit the ground running -...

hit the ground running in second life