k8s in prod
TRANSCRIPT
![Page 1: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/1.jpg)
© 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved.
Kubernetes in PRODVMware SAAS (Cloud Management BU)
Mageshwaran RKumar Gaurav
![Page 2: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/2.jpg)
2
VMware SAAS
1 K8S Usage
2 Dynamic Provisioning
3 Monitoring & Upgrade
VMware SAAS in the current context refers to CMBU initiative alone
![Page 3: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/3.jpg)
3
VMware SAAS: Overview• MCM solution combining the capabilities of Application orchestration, monitoring, management,
costing etc
• SaaS first, platform not a product
• Enables DevOps engineers and Developers to be more agile and leverage resources from any cloud.
• Built using variety of tech Stack– Distributed Micro-services – Java, Scala, Spring, Guice, Xenon etc.,– Relational databases, Key value stores, Document stores etc.,
VMware SAAS in the current context refers to CMBU initiative alone
![Page 4: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/4.jpg)
4
Types of Application
• Green Field (New) Apps– Xenon based Micro-services– Distributed– Soft states often categorized to Stateless apps
• Brown Field (Existing) Apps– Existing applications follows n-tier architecture– Containerization– Leverages K8S Pod – Separate Stateful & Stateless apps
![Page 5: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/5.jpg)
5
Deployments• Core business logic are modelled as stateless containers
– Java & Scala applications– Load based replica sets, auto scale
• Soft state applications– Xenon services & distributed task processing– Distributed applications uses remote nodes to store the data than disk– Spread pod across nodes (anti-affinity)
• Configure Liveness Probe– Health Checks API
![Page 6: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/6.jpg)
6
Stateful Sets• Persistence layer & distributed applications
• Mongo DB– Containerized, replicated– Entry point Initializes and configures replica set– Customized Storage class : aws-ebs (type, iops, zone)
• Postgres DB– Containerized, Active-stand by – Streaming replication– Pgpool
![Page 7: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/7.jpg)
7
Best Practices• Cloud agnostic
– Avoid AWS specific services
• Prefer stateless
• Service Discovery– Internal communication
• Uses FQDN of K8S service name• HTTP, TCP
– External Service communication• NGINX/ HA proxy
• CI/CD Pipeline using vRealize code stream
![Page 8: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/8.jpg)
8
K8S in PROD
1 K8S Usage
2 Dynamic Provisioning
3 Monitoring & Upgrade
![Page 9: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/9.jpg)
9
Dynamic Provisioning
• Use cases – Single instance of N-tier application/distributed application supports “X” tenants– Use case demands due to compliance/ data isolation requirements etc.,– Horizontal scaling of application (collection of heterogeneous K8S resources)
• Onboarding Service– Runs as a deployment inside cluster– Use JSON templates to create K8S resources– Lazy while provisioning and avoid shuffling– Orchestrates updates
![Page 10: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/10.jpg)
10
Dynamic Provisioning
• Control plane– Build using Xenon– K8S REST spec client in Java– Business logic to scale based on the usage & load– Wraps bootstrap logic using containers
• Challenges– K8S Version upgrade– Scale down– Environment abstractions
![Page 11: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/11.jpg)
11
K8S in PROD
1 K8S Usage
2 Dynamic Provisioning
3 Monitoring & Upgrade
![Page 12: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/12.jpg)
12
Monitoring
• Tools Used– VMWare vRealize Log Insight for log monitoring (Kibana is good too)
– Data dog (Grafana is good too)
• Support for all standard processes like JVM, RDBMS, Mongo etc., • Docker & Kubernetes• Support for posting custom metrics
– Pingdom• Users perspective
– Pager duty– Status.io
• Communicate health to stakeholders
![Page 13: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/13.jpg)
13
Monitoring• Application
– Pod contains Data dog agent & Log insight agent– Captures application metrics
• Dropwizard metrics & Xenon Stats• Publishes to data dog
– Custom monitoring dashboards in Data dog
• Dev Ops UI– Uses Kubernetes auth – Separate from application authn & authz
• Fault Injection– Simulates Pod failures– Simulates CPU Usage & network delays
![Page 14: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/14.jpg)
14
Upgrade• Side by side Upgrade
– Preferred option for soft state (deployment) applications– Pauses user request momentarily to minimize downtime
• Rolling upgrade – Preferred option for Stateful sets– Avoid transformations and provide backward compatibility
• Backup & Restore– Soft states are backed up to S3. – EBS are periodically snapshotted to S3– Periodic restore to Staging– DR scenarios
![Page 15: K8S in prod](https://reader036.vdocuments.site/reader036/viewer/2022081505/58ce7e2d1a28ab210a8b4f3f/html5/thumbnails/15.jpg)
Questions ???
15