federation of kubernetes clusters (a.k.a. "ubernetes") - kubecon 2015 slides - quinton...
TRANSCRIPT
![Page 1: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/1.jpg)
Federation of Kubernetes Clusters ("Übernetes")Kubecon 2015
Quinton Hoole <[email protected]>Staff Software Engineer - Googlequinton_hoole@github
![Page 2: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/2.jpg)
Google has beeeg data centers...... but you know that already.
Images by Connie Zhou
![Page 3: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/3.jpg)
But we also have rather a lot of them...
![Page 4: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/4.jpg)
Treating these differently can have benefits...
![Page 5: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/5.jpg)
UI
CLI
API
Control Plane Servers
Kubernetes
Users
containerscontainers
containerscontainers
containers
containerscontainers
containerscontainers
containers
containerscontainers
containerscontainers
containers
Cluster / Data Center / Availability Zone
![Page 6: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/6.jpg)
UI
All you really care about?
API Containers
![Page 7: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/7.jpg)
UI
CLI
API
Control Plane Clusters
Übernetes
API
Users
Kubernetes on
Kubernetes on
Kubernetes on Premise
Federation
![Page 8: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/8.jpg)
Why is this interesting?
![Page 9: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/9.jpg)
Reason 1: High Availability
• Cloud providers have outages, yes, but...• Has one of your application software
upgrades ever gone terribly wrong?• How about infrastructure upgrades
(auth systems? quota? data store?)• How about a fat-fingered config
change?• There are several interesting variants:
• Multiple availability zones?• Multiple cloud providers?
Cross-cluster Load Balancer
Your paying
customer
Cluster 1
Cluster 2
Cluster 3
![Page 10: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/10.jpg)
Reason 2: Application Migration
• Migrating applications between clusters is tedious and error-prone if done manually• Much like software upgrades, you
*can* script them, but (K)ubernetes just does it quicker/safer/better.• Now with rollback too!
• On-premise ↔ Cloud• Amazon ↔ Google :-)• ...
Ubernetes
UI
On-Premise Cluster In-Cloud Cluster
Migrate: On Premise→Cloud
Different Cloud Provider
![Page 11: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/11.jpg)
Reason 3: Policy Enforcement
• Some data must be stored and processed within specified political jurisdictions, by law.
• Some software/data must be on premise and air-gapped, by company policy.
• Some business units get to use the expensive gear, some don't.
• Auditing is also a big deal, so funnelling all operations through a central control point makes this easier.
Ubernetes
UI
U.S. Cloud Cluster E.U Cloud Cluster
On-premise Cluster
![Page 12: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/12.jpg)
Reason 4: Vendor Lock-in Avoidance
• Make it easy to migrate applications between cloud providers.
• Run the same app on multiple cloud providers and choose the best one for your:• workload characteristics• budget• performance requirements• availability requirements
Ubernetes
UI
Kubernetes on GCE Kubernetes on AWS
Kubernetes On-Premise
![Page 13: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/13.jpg)
Reason 5: Capacity Overflow
• Make intelligent placement decisions • Utilization• Cost• Performance Ubernetes
User
On Premise Cluster
Other Cloud Provider
Preferred Cloud Provider
Run my stuff
![Page 14: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/14.jpg)
"OK, I'm sold. Where's the catch?"
![Page 15: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/15.jpg)
Provider 1
Zone A
Zone B
Federation comes with some challenges...
Provider 2Zone C
Provider 1
Zone D
● Different bandwidth charges/latency/through-put/reliability
● Different service discovery (but DNS!)
● Consolidated monitoring & alerting
![Page 16: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/16.jpg)
Cross-cluster load balancing
• Geographically aware DNS gets clients to the "closest" healthy cluster.
• Standard Kubernetes service load balancing within each cluster.
• New L7 LB's available soon.• Can be extended to divert traffic away from
"healthy-but-saturated" clusters.
![Page 17: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/17.jpg)
Cross-cluster service discovery
• DNS + Kubernetes cluster-local service discovery.
• Can default to cluster-local with failover to remote clusters.
![Page 18: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/18.jpg)
Location affinity
• Strictly coupled pods/applications• High bandwidth requirements• Low latency requirements• High fidelity requirements• Cannot easily span clusters
• Loosely coupled• Opposite of above• Relatively easily distributed across
clusters• Preferentially coupled
• Strongly coupled but can be migrated piecemeal.
![Page 19: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/19.jpg)
Cross-cluster monitoring and auditing...
• "Cluster per tab" might suffice for small numbers of clusters
• Some monitoring solutions provide stronger integration and global summarization
![Page 20: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/20.jpg)
Cluster Federation - The Implementation...
![Page 21: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/21.jpg)
API Compatible with Kubernetes
• Less new stuff to learn• Can learn incrementally, as you
need new functionality.• Analogous argument applies to
existing automation systems (PAAS etc). • These can be ported to
Ubernetes relatively easily.• All Kubernetes entities are
"federatable".
Ubernetes or Kubernetes
Client
Applications
Applications
Applications
Run my stuff
![Page 22: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/22.jpg)
State and control resides in underlying clusters (for the most part)
• Better scalability• Kubernetes scales with
number of nodes per cluster (<10,000)
• Ubernetes scales with number of clusters (~100)
• Beter fault isolation• Kubernetes clusters fail
independently of Ubernetes
Kubernetes Cluster Kubernetes Cluster
Ubernetes
API
APIRepl. Ctrl etcState
API
APIRepl. Ctrl etcState
API
APIRepl. Ctrl etcState
![Page 23: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/23.jpg)
• Drive current state -> desired state• But per-cluster state, not per node,
per pod etc.
• Observed state is the truth
Recurring pattern in the system
Examples: • ReplicationController• Service
observe
diff
act
Similar Control loops to Kubernetes
![Page 24: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/24.jpg)
Modularity
Loose coupling is a goal everywhere• simpler• composable• extensible
Code-level plugins where possible
Multi-process where possible
Isolate risk by interchangeable parts
Examples:• MigrationController• Scheduler
![Page 25: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/25.jpg)
Federation status & plans
Federation Lite (single cluster, multiple zones)• In alpha Q4 2015• Productionized ~Q1 2016
Federation Proper (multiple clusters, federated)• Alpha Q1 2016
Google Container Engine (GKE)• hosted Federation too• GKE Federation Lite ~Q1-Q2 2016
PaaSes and Distros• RedHat OpenShift, CoreOS Tectonic, RedHat Atomic...• ... watch this space...
![Page 26: Federation of Kubernetes Clusters (a.k.a. "Ubernetes") - KubeCon 2015 slides - Quinton Hoole](https://reader031.vdocuments.site/reader031/viewer/2022021813/5876fd5c1a28abf3398b6a03/html5/thumbnails/26.jpg)
I want more!
• Requirements doc - comments welcome• tinyurl.com/ubernetesv2
• Special interest group• groups.google.com/forum/kubernetes-sig-federation
• [email protected]• quinton_hoole@github
Kubernetes Cluster Kubernetes Cluster
Ubernetes
API
APIRepl. Ctrl etcState
API
APIRepl. Ctrl etcState
API
APIRepl. Ctrl etcState