Deploying stateful apps and multi-cluster federation with K8s 1.5

Agenda

1

2

3

4

5

Introduction

Support of stateful apps with StatefulSets

Cluster of Clusters - Kubernetes Federation

2

Q & A

Networking

DoIT International confidential │ Do not distribute

About us..

Vadim SoloveyCTO

Doron OffirDirector of Engineering

DoIT International confidential │ Do not distribute

DoIT International confidential │ Do not distribute

DoIT International confidential │ Do not distribute

Stateful Sets

StatefulSets (aka PetSets)

Support for stateful applications

Stateless Apps/Containers

Deployments and ReplicaSets are a great way to run stateless replicas of an application on Kubernetes, but their semantics aren’t right for deploying stateful applications.

StatefulSets

StatefulSets are intended to be used with stateful applications and storage-bound applications.

Features:● Ordered Creation of Pods● Ordered Termination● Unique Network Identities / DNS names● Persistent Stable Storage

Questions to ask yourself:

● Remote Storage vs Local Storage?● Do I need to Scale storage application?● Do I absolutely need to squeeze the absolute maximum QPS from my storage subsystem?● Can I run my workload on unified instances rather than specialized hardware?

Pod Disruption Budget

Pod Disruption Budget is a safety guard/constraint on pods.

● Voluntary evictions● Non-Voluntary evictions

Pod Disruption Budget = minimum number or percentage of replicas of a collection that

must be up at a time surviving voluntary evictions

StatefulSets 101

kind: StatefulSetmetadata: name: zkspec: serviceName: zk-headless replicas: 3 spec: containers: - name: k8szk imagePullPolicy: Always image: gcr.io/google_samples/k8szk:v1 ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election command: - sh - -c - zkGenConfig.sh && zkServer.sh start-foreground readinessProbe: exec: command: - "zkOk.sh" initialDelaySeconds: 15 timeoutSeconds: 5

kind: PodDisruptionBudgetmetadata: name: zk-budgetspec: selector: matchLabels: app: zk minAvailable: 2

spec: serviceName: zk-headless replicas: 3 annotations: pod.alpha.kubernetes.io/initialized: "true" scheduler.alpha.kubernetes.io/affinity: > { "podAntiAffinity": { "requiredDuringSchedulingRequiredDuringExecution": [{ "labelSelector": { "matchExpressions": [{ "key": "app", "operator": "In", "values": ["zk-headless"] }] }, "topologyKey": "kubernetes.io/hostname" }] } }

Demo: Running ZooKeeper with StatefulSets

Demo flow:

● Creating a ZooKeeper Ensemble (zk)

● Check identifier for each pod to allow proper leader election thru ZAB protocol

● Check the FQDN of each pod in the zk StatefulSet & have correct zoo.cfg

● Sanity Testing the Ensemble

● Verifying we have durable storage

● Ensuring Consistent Configuration using ConfigMaps

● Configuring Logging & working with non-privileged users using securityContext

● Managing the ZooKeeper Process, Liveness and Readiness Probes

● Tolerating Node Failure

● Surviving Maintenance using PodDisruptionBudget

Federation

Clusters Federation

Clustering the clusters

Federation

Taking the management of k8s services one step forward

Federation aims to allow us a simple central management of our k8s service across multiple clusters, regardless of underlying infrastructure or vendor

Version 1.5 support additional resources at the Federation level and now support:● ConfigMap● DaemonSet● Deployment● Event● Ingress● Namespace● ReplicaSet● Secret● Services

Federation under the hood

One of the Kubernetes clusters must become a master by running a Federation Control Plane.

In practice, this is a controller that monitors the health of other clusters, and provides a single entry point for administration. The entry point behaves like a typical Kubernetes cluster.

The Federation Control Plane components consist of:Components as pods managed by Deployments on your elected Kubernetes cluster. It also starts a type: LoadBalancer Service for the federation-apiserver and a PVC backed by a dynamically provisioned PV for etcd. All these components are created in the federation namespace.

Those allow us to send API requests to the Federation endpoint, for example creating a service, which will result in that service been created across all federated clusters.

$ kubectl --context=federation create -f rs/k8shserver

Federation - Kubefed

Version 1.5 added Kubefed to our arsenal, making the creation of Federation easier.

To obtain it you can:Source article - https://kubernetes.io/docs/admin/federation/kubefed/#getting-kubefed

$ curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.0/kubernetes-client-linux-amd64.tar.gz

$ tar -xzvf kubernetes-client-linux-amd64.tar.gz

$ sudo cp kubernetes/client/bin/kubefed /usr/local/bin

$ sudo chmod +x /usr/local/bin/kubefed

$ sudo cp kubernetes/client/bin/kubectl /usr/local/bin

$ sudo chmod +x /usr/local/bin/kubectl

Federation hands-on example

The following example is based on Kubernetes article Cluster Federation in Kubernetes 1.5

We will make use of the following scripts

In this example we will: ● Setup Federation on one cluster and add two other clusters to the Federation.● Deploy some components:

○ Replica Set○ Service○ Ingress

Federation hands-on example

1. Create clusters using gcloud container clusters create:gcloud container clusters create gce-us-east1-b --cluster-version=1.5.1 --project=${FED_PROJECT} --zone=us-east1-b --scopes cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite

2. Authenticate to clusters using gcloud container clusters get-credentials:gcloud container clusters get-credentials gce-us-east1-b --zone=us-east1-b --project=${FED_PROJECT}

3. Setup Federation plane using kubefed init federation:kubefed init federation --host-cluster-context=gke_${FED_PROJECT}_${FED_HOST_CLUSTER}_gce-${FED_HOST_CLUSTER} --dns-zone-name=${FED_DNS_ZONE}

4. Join clusters to Federation using kubefed join:kubefed --context=federation join cluster-europe-west1-b --cluster-context=gke_${FED_PROJECT}_europe-west1-b_gce-europe-west1-b --host-cluster-context=gke_${FED_PROJECT}_${FED_HOST_CLUSTER}_gce-${FED_HOST_CLUSTER}

5. Deploy our app