#vmworld
CODE1360U
Kubernetes Operatorsfor VMware Enterprise PKS and VMware Cloud PKS
Tom Schwaller, VMware, Inc.Michael Gasch, VMware, Inc.
#CODE1360U
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U
Disclaimer
This presentation may contain product features or functionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
2
The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U
Agenda
3
Kubernetes Operators
Introduction
Custom Resource Definitions (CRDs)
CRD Examples (CronTab, PKS Sink Resources)
Operators and Operator Frameworks
Controller Fundamentals
Architecture
Writing Controllers
Kopf Demo
Kopf (Kubernetes Operator Pythonic Framework)
VM-Operator using Kopf
VMworld 2019 Content: Not for publication or distribution
4©2019 VMware, Inc.
Kubernetes Operators
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 5
November 03, 2016 - CoreOS Blog Post by Brandon Philips Introduced and described the Concept of a Kubernetes Operator
Published two Examples: etcd and Prometheus Operator
Introduction
Kubernetes Operators
An Operator is an application-specific controller thatextends the Kubernetes API to create, configure, andmanage instances of complex stateful applications on behalf of a Kubernetes user.
Custom Resource Definition Custom ControllerOperator
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 6
Kubernetes Objects defining Custom Object Types
New API Endpoints
Example: Crontabs
/apis/stable.example.com/v1/namespaces/*/crontabs/...
Introduced in Kubernetes 1.7
Scope: Namespace or Cluster
Once registered you can create Objects of that Type
OpenAPI 3.0 Validation Schema
Beta in Kubernetes 1.15
Allows CRD Pruning: Deletion of unknown fieldsspec.preserveUnknownFields: false
Custom Resource Definitions (CRDs)
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
Extending the Kubernetes API
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 7
Example: CronTab CRD + Custom Resource CreationapiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
preserveUnknownFields: false
validation:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
image:
type: string
replicas:
type: integer
default: 1
$ cat my-crontab.yaml
apiVersion: "stable.example.com/v1“
kind: CronTab
metadata:
name: my-cron-object
spec:
cronSpec: "* * * * */5"
image: my-cron-image
replicas: 1
$ kubectl apply -f my-crontab.yaml
$ kubectl get crontabs
NAME AGE
my-cron-object 6s
using
CR
CRD
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 8
Enterprise PKS - Sink Resources (CRDs)
https://docs.pivotal.io/runtimes/pks/1-4/create-sinks.html
Global Logging
Cluster Logging
Namespace Logging
overwrites
overwrites
Enable PKS users to configure Log-Destinations
Supported Protocols: Syslog (RFC 5424)
Log-Forwarding per Cluster or per Namespace
Using fluentbit Log Aggregator
Cluster & Namespace Sinks implemented as K8s CRDs
sinks.apps.pivotal.io
clustersinks.apps.pivotal.io
Enabled/disabled globally (i.e. for all Clusters) in PKS Tile
Specified by YAML file
kubectl create -f sink.yaml
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 9
Sink Resources - Cluster View
K8s Cluster
Worker
sink-controller
Worker
fluentbit
Master Worker
fluentbit
Worker
fluentbitfluentbit
elastic stackLog Insight splunk>
The sink-controller and fluentbit Pods belong to the Namespace pks-system.VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 10
New Custom Ressource Definitions, sink-contoller Deployment and fluentbit Daemonset
Sink Resources in PKS Cluster
root@cli-vm:~# kubectl get crd
clustersinks.apps.pivotal.io 2018-11-27T15:57:03Z
sinks.apps.pivotal.io 2018-11-27T15:57:03Z
kubectl get all –n pks-system
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 11
Sink Resources - Worker Node View
Worker Node
Namespace A
Pod 2Pod 1
Namespace B
Pod 2Pod 1
Namespace C
Pod 2Pod 1
Pod stdout & stderr
Pod stdout & stderrnamespace
Pod stdout & stderr
elastic stack
Log Insight
pks-system
fluentbit
The fluentbit Pod on each Worker Node aggregates Logs from
all Namespaces and forwards them to the corresponding endpoint.VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 12
Kubernetes Operators
https://kubedex.com/operators/https://operatorhub.io/
A Selection
Operator Domain URL
Velero Backup https://velero.io/
Kanister Backup https://github.com/kanisterio/kanister
Tensorflow Big Data https://github.com/kubeflow/tf-operator
Jaeger Tracing https://github.com/jaegertracing/jaeger-operator
MongoDB Database https://github.com/mongodb/mongodb-enterprise-kubernetes
Spinnaker CI/CD https://operatorhub.io/operator/spinnaker-operator
Minio Storage https://github.com/minio/minio-operator
OpenFaaS FaaS https://github.com/openfaas-incubator/openfaas-operator
Kafka Streaming https://github.com/strimzi/strimzi-kafka-operator
Aqua Security https://github.com/aquasecurity/aqua-operator
Istio Service Mesh https://github.com/banzaicloud/istio-operator
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 13
Operator Framework - https://github.com/operator-framework
Operator SDK (released Mid 2018 by Red Hat)
Operator Lifecycle Manager
Operator Metering
Supported Languanges: Golang
• You can also use Helm Charts or Ansible Playbooks
Kubebuilder - https://github.com/kubernetes-sigs/kubebuilder
Supported Languages: Golang
Kopf - https://github.com/zalando-incubator/kopf
Zalando Open Source Project
Supported Languages: Python
Google Metacontroller - https://metacontroller.app
Google Open Source Project
Supported Languages: Python, JavaScript, etc.
Kubernetes Operator Toolkits
VMworld 2019 Content: Not for publication or distribution
14©2019 VMware, Inc.
Kubernetes ArchitectureFrom the View of a Controller Developer
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 15
Overview
Kubernetes Architecture
Co
ntr
ol P
lan
e
API Serveretcd
ControllerManager
Scheduler …
Acce
ss
REST API SDKs Web UI kubectl
Wo
rke
rs
Kubelet Kubelet Kubelet
Pod Pod Pod
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 16
In Simple Terms…
Kubernetes Architecture
Co
ntr
ol P
lan
e
API Server
Wo
rke
rs
= Control LoopVMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 17
Request Flow
Kubernetes Architecture
API ServerREST
REST
DecodingConversion &
DefaultingAdmission
Persistency (etcd)
…
WA
TC
HP
OS
T
$ kubectl create –f my_replicaset.yaml
apiVersion: extensions/v1beta1
kind: ReplicaSet
spec:
replicas: 2
Commands
Events
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 18
Commands and Events
Kubernetes Architecture
Commands Events
• Requests (intent) to do something
• Named in the imperative, e.g. “CREATE”
• Can be rejected
• Typically used in synchronous 1-to-1 request/responsecommunication
• Higher coupling between sender and receiver
• Something that has happened (a fact)
• Named in past tense, e.g. “CREATED”
• Cannot (semantically) be rejected by receiver
• Asynchronous 1-to-many communication, e.g. publish/subscribe
• Low coupling between sender and receiverVMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 19
Choreography, not Orchestration
Kubernetes Architecture
AP
I S
erv
er
CREATE
apiVersion: extensions/v1beta1
kind: ReplicaSet
spec:
replicas: 2
CREATEPod
ReplicaSetCREATED
ReplicaSetController
BINDPod
PodCREATED
Scheduler Kubelet
PodBOUND
UPDATEPod
(“running”)
Time
Command
EventVMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 20
Control Loop (Reconciliation)
Kubernetes Architecture
Observe
Analyze
Act
apiVersion: extensions/v1beta1
kind: ReplicaSet
spec:
replicas: 2
desired := getDesiredState() current := getCurrentState()
diff := desired – current
if diff < 0 {
deletePods()
}
if diff > 0 {
createPods()
}
Command
Event
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 21
Controllers, oh my…
Kubernetes Architecture
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 22
Further Reading
Kubernetes Architecture
VMworld 2019 Content: Not for publication or distribution
23©2019 VMware, Inc.
Writing ControllersGuidelines and Principles
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 24
Kubernetes comes with several features to make the life of a (controller) developer easier
Scheduling and Supervision (self-healing*)
Configuration and Secret Management
Service Discovery and Networking
Storage Management
(Cloud) Portability
Declarative API Stability and Extensibility (CRDs)
AuthN and AuthZ (RBAC)
SDKs
(Probably a lot more than we could fit on this slide)
* Self-healing (eventual recovery) != HA
Net Benefits Working With Kubernetes
Writing Controllers
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 25
Depending on the complexity of your controller, there might be a steep learning curve
Lots of primitives and objects to learn
client-go (de facto SDK) “is not for mere mortals” (Bryan Liles)
Optimistic concurrency in an asynchronous eventual consistent system
There is No Now
The (global) state is always behind you (distributed, delayed and unknown to the local observer)
Fast moving project
BUT!
Writing Controllers
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 27
Writing ControllersRequired Mindset
Single Responsible Principle
Decoupling via event-driven messaging
No central coordinator
Eventual consistent by
design
Don’t rely on (assume) order
API server (etcd) is the source of
truth*
In-memory cache via reconciliation
Things will go wrong (crash)
No shared (wall) clock
Anticipate effects on the rest of the
system
Delivery and processing
guarantees only within Kubernetes
AutonomousProcesses
Concurrency&
Asynchrony
Statelessover
Stateful
Side EffectsDefensive
Programming
VMworld 2019 Content: Not for publication or distribution
28©2019 VMware, Inc.
Kopf Demo
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 29
Easy Way to write Kubernetes Operators
Kopf - Kubernetes Operator Pythonic Framework
https://kopf.readthedocs.io/en/latest/https://bit.ly/2KoLvmP
$ kopf run example.py$ kubectl apply -f obj.yaml
$ kubectl get KopfExample kopf-example-1 -o yaml
apiVersion: zalando.org/v1
kind: KopfExample
metadata: ...
spec:
duration: 1m
field: value
items:
- item1
- item2
status:
message: hello world
# Custom Resource for Kopf Example Operator
apiVersion: zalando.org/v1
kind: KopfExample
metadata:
name: kopf-example-1
labels:
somelabel: somevalue
annotations:
someannotation: somevalue
spec:
duration: 1m
field: value
items:
- item1
- item2
obj.yaml example.pyimport kopf
@kopf.on.create('zalando.org', 'v1', 'kopfexamples')
def create_fn(spec, **kwargs):
print(f"And here we are! Creating: {spec}")
return {'message': 'hello world'} # will be new status
1 2
3
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 30
Conceptual Overview
Kopf vSphere VM Operator
vSphere
VMTemplate
Kubernetes
Folder:Example-1
VM VM
VM VM
Folder:Example-2
VM VM
ESXi Cluster
CRExample-2
CRExample-1
API ServerKopf
VM Operator
LIST/WATCH()
vCenter
CREATE()CLONE()
DESTROY()
VMworld 2019 Content: Not for publication or distribution
32©2019 VMware, Inc.
DEMO
https://github.com/embano1/kopf-operator-vmworld
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 33
Example: etcd Operator (1)Installation, Load Balancer Configuration
$ git clone https://github.com/coreos/etcd-operator.git
$ cd etcd-operator/
$ ./example/rbac/create_role.sh Create RoleBinding for the etcd Operator$ kubectl apply -f ./example/deployment.yaml Create etcd Operator$ kubectl apply -f ./example/example-etcd-cluster.yaml Create 3-node etcd Cluster
$ cat example/example-etcd-client-service-lb.yaml Create LoadBalancer for etcd-ClusterapiVersion: v1
kind: Service
metadata:
name: example-etcd-client-service-lb
namespace: default
spec:
ports:
- name: client
port: 2379
protocol: TCP
targetPort: 2379
selector:
etcd_cluster: example-etcd-cluster
type: LoadBalancer
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
name: "example-etcd-cluster"
spec:
size: 5
version: "3.3.13"
To resize/upgrade the etcd Cluster change its properties and re-apply
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 34
Example: etcd Operator (2)Connnecting to the etcd Cluster
$ kubectl get svc | grep example-etcd-client-service-lb | awk '{print $4;}‘
10.40.14.41,100.64.48.5
$ export ETCDCTL_API=3
$ export ETCDCTL_ENDPOINTS=http://10.40.14.41:2379
$ wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz
$ tar xvfz etcd-v3.3.13-linux-amd64.tar.gz
$ sudo mv etcd-v3.3.13-linux-amd64/etcdctl /usr/local/bin
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
etcd-operator-85f7494fcf-4xql5 1/1 Running 0 69m
example-etcd-cluster-5kkp8lxrsn 1/1 Running 1 51m
example-etcd-cluster-6bbnlbp7kb 1/1 Running 1 67m
example-etcd-cluster-f2gzqtmh57 1/1 Running 1 51m
$ etcdctl put /message "Hello World"
$ etcdctl get /message
/message
Hello WorldVMworld 2019 Content: Not for publication or distribution
35©2019 VMware, Inc.
Resources
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 36
Kopf vSphere VM Operator Github Repository
• https://github.com/embano1/kopf-operator-vmworld
Zalando Kopf Github Repository:
• https://github.com/zalando-incubator/kopf
Awesome Operators
• https://github.com/operator-framework/awesome-operators
Programming Kubernetes (Book)
• https://learning.oreilly.com/library/view/programming-kubernetes/9781492047094/
Links and Resources (1)
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 37
Kubernetes API Reference (v1.15)
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/
Kubernetes Sample Controller (CRD)
• https://github.com/kubernetes/sample-controller
Writing Controllers
• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md
Internals of how Controllers are implemented under the Covers
• https://medium.com/@cloudark/kubernetes-custom-controllers-b6c7d0668fdf
Kubernetes Operator Best Practices
• https://blog.openshift.com/kubernetes-operators-best-practices/
Links and Resources (2)
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 38
KubeCon Sessions (Controllers and Operators)
• Writing Kube Controllers for Everyone
– https://www.youtube.com/watch?v=AUNPLQVxvmw
• Writing Kubernetes Controllers for CRDs: Challenges, Approaches and Solutions
– https://www.youtube.com/watch?v=7wdUa4Ulwxg
• How We Built Contour, and What You Can Learn From Our Experience
– https://www.youtube.com/watch?v=4usXJE0EwHo
• Keep the Space Shuttle Flying: Writing Robust Operators
– https://www.youtube.com/watch?v=uf97lOApOv8
• Controllers: Lambda Functions for Extending your Infrastructure
– https://www.youtube.com/watch?v=TM-2GgQ6Q2A
Links and Resources (3)
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. #CODE1360U 39
KubeCon Sessions (Architecture)
• The Kubernetes Control Plane for Busy People Who Like Pictures
– https://www.youtube.com/watch?v=zCXiXKMqnuE
• The Magic of Kubernetes Self-Healing Capabilities
– https://www.youtube.com/watch?v=91dgNqma7-Q&t=2s
• Kubernetes Design Principles: Understand the Why
– https://www.youtube.com/watch?v=ZuIQurh_kDk&t=1s
Links and Resources (4)
VMworld 2019 Content: Not for publication or distribution
VMworld 2019 Content: Not for publication or distribution
TPM, CNABU Platform Architect (OCTO)@tom_schwaller @embano1
Tom Schwaller Michael Gasch
VMworld 2019 Content: Not for publication or distribution