deep dive: vmware kubernetes engine-k8s as a service on ... · deep dive: vmware cloud pks k8s as a...

#vmworld

Deep Dive: VMware Cloud PKS

K8s as a service on Public Cloud

Valentina AlariaDirector of Product Management, VMware, Inc.

Tom SpoonemoreProduct Manager, VMware, Inc.

CNA3124BE

#CNA3124BE

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.



VMware Kubernetes Engine is now …


Agenda


VMware Cloud PKS: Intro & Architecture

VMware Smart Cluster: Intro & Deep Dive

Connections

Policy Framework

Demo

Q&A



Kubernetes makes it easy to manage applications that are container-based.

Easy to deploy your application• Applications are composed of pods that are sets of containers running on the same host• Applications are easily spread across multiple nodes

Easy to scale your application• You can easily increase/decrease the number of instances of your application• You can automate the scaling of your application easily

Easy-to-use network connectivity• Kubernetes makes it easy to deploy load balancers in front of your application• Network policies allow protection between applications deployed in a cluster

Easy-to-use storage• Persistent volumes can be dynamically allocated and used by your application

A quick reminder: What is Kubernetes?



Kubernetes Topology:• A set of master nodes manage the cluster• A set of worker nodes run your application

Kubernetes applications:• Pod: a container (or set of containers) run on one

nodeThink: One instance of your application

• ReplicaSet: A set of identical pods:Think: Your application

• Service: Acts like a load balancer for a set of podsCan be internal (between pods)Can be external (allow access from clients)

Kubernetes

Master Master Master

Worker Worker Worker

Service: Your app

Pod Pod Pod



VMware’s Kubernetes PortfolioMeeting customers wherever they run their apps on any infrastructure

Enterprise Software Cloud Service

PublicBeta

VMware Cloud PKS VMware PKS



US West

US East

Europe West

An enterprise-grade Kubernetes-as-a-Service offering in the VMware Cloud Services portfolio.

• Pay by the second, on-demand over public internet

• Launching on multiple AWS regions, with support for Azure and other platforms in the future

• Globally consistent policy management

• Certified Kubernetes conformant• Full integration with AWS services

VMware Cloud PKSHighly Secure and Available Kubernetes Service



VMware Cloud PKS Part of VMware Cloud Services, Offered as a SaaS-based Model

9

VMware Cloud Services (cloud.vmware.com)

• Single Sign-on

• Single Bill to Manage

• Single Global Support

VMware Cloud PKS

Enterprise grade Kubernetes-as-a-Service offering that provides easy to use, secure by default, and highly efficient containers.


VMware Cloud PKS: Production-Grade K8s for Enterprise

Easy to Use and Maintain

Highly Secure by Default

Integrates with EcosystemMulti-tenancy

• Multi-AZ Master and Etcd

• 7x24 Continuous Health Monitoring of entire cluster

• Auto-remediation of all issues

Highly Available “Dial Tone” Kubernetes

• Hardened Kubernetes

• Continuously patched

• Data encrypted in transit and at rest

• Role and resourced based access policies

• Each customer in isolated AWS, Azure account

• Powerful policy framework across all cloud platforms and all regions

• 90 seconds from account activation to first cluster

• No training or staffing required

• Fully tested and documented integrations with Jenkins, Prometheus, Istio and others

• Integration with AWS services over private networkVMworld 2018 Content: Not for publication or distribution


Kubernetes Ecosystem

Native Kubernetes compatibility with leading open-source solutions and tools

Commercial Partners & Solutions

Broadening feature portfolio & solutions for application deployment

VMware Cloud Services

Simple and easy integration with other VMware Cloud Services

VMware Cloud PKS: Integration EcosystemBroad Ecosystem of Application Building Blocks

WavefrontCode Stream



• Service Meshes: microservice network layer for handling service-to-service communication

• Declarative policies for application observability, traffic management, and security functionalities

• Istio brings enhanced set of L7 metrics used for application monitoring, logging and tracing by Operators & Users

• Istio powers advanced traffic routing and management• Canary Deployments• Rate Limiting• Circuit Breakers• Authentication

Announcing Service Mesh & ObservabilityVMware Cloud PKS Validated Solution: Istio



VMware Cloud PKS Architecture



VMware Cloud PKS Architecture

AWS: us-west-2

Clusters

AWS: eu-west-1

Clusters

Azure: West US

Clusters

VMware Cloud PKS Management Plane

Fully distributed, highly available management plane• Each region shares config data but operates independently• Deployed across multiple availability zonesIsolated, highly available clusters• Each cluster within a single region, spread across availability zones• Each cluster isolated within private network• Each customer isolated within distinct cloud provider account, managed by VMware Cloud PKS

…



Smart Cluster Introduction



VMware Smart Cluster

VMware Smart Cluster automates selection of compute resources constantly optimizing resource usage and reducing cost.

• Removes need for educated guesses around cluster definition and sizing

• Enables management of cost-effective, scalable Kubernetes clusters that are constantly optimized to application needs.

• Provides built-in resiliency with routine health checks and self-healing capabilities for Kubernetes clusters.

• Makes it seamless for a user to run and/or manage highly availabledeployments without additional cost and complexity.

VMware Smart Cluster Run Kubernetes without Managing Servers or Clusters


©2018 VMware, Inc. 17

Demo



Smart Cluster Deep Dive



Production ClusterDetail View

Namespaces vke-system kube-system default

Pods(vke-system)

Smart cluster monitor Ingress CNI

Availability Zone A

Master Node

+etcd

Worker Nodes

Availability Zone B

Availability Zone C

Master Node

+etcd

Worker Nodes

Master Node

+etcd

Worker Nodes

Namespaces vke-system kube-system default

Pods(vke-system)


Availability Zone A

Master Node

+etcd

Worker Nodes

Availability Zone B

Availability Zone C

Master Node

+etcd

Worker Nodes

Master Node

+etcd

Worker Nodes

Production Cluster 1

• Clusters in managed, isolated AWS account

• Each cluster is in a single region

• Each cluster is in an isolated VPC

• Clusters are fully HA:

• Multiple masters

• Three availability zones

Production Cluster 2



Development ClusterDetail View

• Clusters in managed, isolated AWS account

• Each cluster in a single region

• Shared VPC for multiple Development Clusters

• Developer sandbox or test environments

• Single Master node per cluster

Namespaces vke-system

kube-system default

Pods(vke-

system)


Master Node

+etcd

Worker Nodes

Worker Nodes

Worker Nodes

Availability Zone

Namespaces vke-system

kube-system default

Pods(vke-

system)


Master Node

+etcd

Worker Nodes

Worker Nodes

Worker Nodes

Dev Cluster 1 Dev Cluster 2



• Smart cluster monitor watches pods and nodes and contacts VMware Cloud PKS management plane when changes are needed.

• When a pod is created and cannot be scheduled, VMware Cloud PKS creates a new worker node.

• When pods are deleted and nodes are underutilized, VMware Cloud PKS deletes worker nodes and pods are moved as necessary

ElasticityElastic Smart Clusters

Smart Cluster

Master Worker Worker

Developers

VMware Cloud PKS

Smart Cluster Monitor

Kubernetes Scheduler

Kubernetes Pod Pending

Worker

Deployments

Kubernetes API

Pods

Clo

ud A

PI



• Managed OS patching for Kubernetes nodes

• On demand upgrades for Kubernetes clusters

OS Patches and Kubernetes Upgrades

Automated PatchingKubernetes Nodes OS Patching

What• Patches will be applied within 7

days from the time the patch is available for all Critical Vulnerability CVE’s >9

• 14 days for CVE’s >7.5 and <9

How• Patches applied automatically by

VMware Cloud PKS• Nodes rebooted one at a time• Production Smart Clusters have zero-

downtime because there are three masters.

Kubernetes Upgrade What

• Patch Releases: 2 weeks after release• Minor Releases : 4 weeks after release• Major Releases: TBD

How• Kubernetes upgrades initiated by customers• Rolling updates applied to cluster• Master nodes get upgraded first• etcd’s are backed up before upgrading• Cluster is rolled back if upgrade fails



- Data Encryption at rest and in Motion

- Encrypted Kubernetes secrets in etcd

Encryption

Data Encryption

VM

war

e C

loud

PK

S

User profile

Application data

Policy dataUsers

Applications

Https

Https

- TLS Encryption for traffic- From Users to VMware Cloud PKS- Between masters, worker and etcVMworld 2018 Content: Not for publication or distribution


Health Monitoring

Each Smart Cluster actively monitored• Capacity• Faults

etcd Kubernetes API Server

Kubernetes Scheduler

Kubernetes UI

# of Master and Worker

nodes

Smart Cluster Monitor

Smart Cluster Scaler

K8s Dashboard KubeDNS CNI

Manager

VMware Cloud PKS Service Pods

Kubernetes Cluster

Health Metrics Stream

Metrics

VMware Cloud PKS Ops Team Remediation

BotsVMworld 2018 Content: Not for publication or distribution


Smart Cluster Summary

Features Development Smart Cluster

Production Smart Cluster

Pod Networking X X

Elasticity X X

Managed OS Upgrades X X

Health Monitoring X X

Validated Solutions X X

Zero-Downtime Upgrade X

VPC Network Isolation X

High Availability X

Connections X



Connections



AWS Service Access using VPC Peering

Customer Deployed Virtual Private Cloud (VPC) Assets

EC2 Instance Applications

Private Load Balancer Listeners

Proxied Services

AWS PrivateLink Endpoint Services

Peer VPC Private DNS (in-region peers only)

Amazon EC2

Amazon VPC

AmazonS3

Amazon EFS

AmazonRDS

AmazonDynamoDB

AmazonRoute 53

AWS Service Ecosystem

And many others …



Customer AWS

Account

Cloud PKS AWS

Shadow Account

Connecting your existing VPCSimple Access to AWS Services

Subnet :10.16.1.0/24VPC

AWS A/C: cpks-shadow-tenant-A

VPC peering

Subnet:.10.0.0.0/24

VPC

AmazonRDS

AWS A/C: Peer User Account

Your Existing AWS Account and VPC• Self-managed Deployment• Integrated AWS Services

Your Cloud PKS Account• Cloud PKS managed VPC• Production Smart Cluster

Connections initiated VPC Peer



Features

• L3 Connectivity to other AWS Account VPCs

• Supports cluster egress traffic today

• Traffic routed internally on AWS private networks

• Security Groups protects cluster from unauthorized access

• Private DNS name resolution supported within region

• Traffic charged at inter-AZ or inter-region rates

Considerations

• VPC IP CIDR blocks cannot overlap

• Max of 50 active peering connections / VPC

What to know about VPC Peering?

Customer AWS

Account

Cloud PKS AWS

Shadow Account

Subnet :10.16.1.0/24VPC

AWS A/C: cpks-shadow-tenant-A

VPC peering

Subnet:.10.0.0.0/24

VPC

AmazonRDS

AWS A/C: Peer User Account



Demo



Policy Framework



Access Policy Framework

CONFIDENTIAL

Organization

…

Folder 1 Folder 2

Project 1 Project 2 Project 3

Cluster 1 Cluster 2

Namespace 1 Namespace 2

Cluster X

Namespace Y

Each customer is mapped to an organization.

An organization is hierarchical.

Access policies are applied anywhere, and inherited down the hierarchy.

Access policies are pushed to Kubernetes cluster and enforced within Kubernetes

VMware Cloud PKS Logical Resources

Kubernetes ResourcesVMworld 2018 Content: Not for publication or distribution


Access Policy FrameworkAccess Policy & Defaults

• Access Policies can be defined at any node of the hierarchy tree

• All Users by default have access to “Shared Folder” and “Shared Project” in each Organization

• By default the Organization has below policy

Node Role User/Group

Org *.iam.edit, *.edit, smartcluster.admin

VKEServiceAdministrators

Org Org.view, Group.view

VKEServiceUsers

Shared Folder

Folder.view VKEServiceUsers

Shared Project

Project.edit VKEServiceUsers

Shared Project

SmartCluster.admin VKEServiceUsers

Organization

…

Folder 1 Shared Folder

Project 1 Project 2 Shared Project

Cluster 1 Cluster 2


Cluster X

Namespace Y



Granting Alice the “SmartCluster.Edit” role on a folder gives her SmartCluster.Edit on all clusters within that folder.

Access Policy Example #1Enabling a user to access all clusters in a folder

Organization

…

Folder 1

Edit:Alice

Folder 2


Cluster 1

Edit:Alice

Cluster 2

Edit:Alice


Cluster X

Namespace Y

Alice is granted accessto all clusters in folder 1

Customer’s Logical View



Granting Bob access in a project gives him access to all clusters in that project.

Access Policy Example #2: ProjectEnabling a user to access all clusters in a project

Organization

…

Folder 1 Folder 2


Cluster 1 Cluster 2


Cluster X

Namespace Y


Bobis granted accessto all clusters in Project 1

Edit:Bob Edit:Bob



Granting Carol access to a single cluster gives her access to only that cluster.

Access Policy Example #3: ClusterEnabling a user to access just one cluster

Organization

…

Folder 1 Folder 2


Cluster 1 Cluster 2


Cluster X

Namespace Y


Edit:Carol

Carol is granted accessonly to cluster 1



Granting David access to a single namespace only grants access to that single namespace in a single cluster.

Access Policy Example #4: NamespaceEnabling a user to access just one namespace in one cluster

Organization

…

Folder 1 Folder 2


Cluster 1 Cluster 2


Cluster X

Namespace Y


David is granted accessonly to one namespace

in Cluster 1

Edit:David



Demo



Visit cloud.vmware.com Request Access Log onto console.cloud.vmware.com

and start using service

Getting Started with VMware Cloud Services is Easy


DON’T FORGET TO FILL OUT YOUR SURVEY.

#vmworld #CNA3124BE


THANK YOU!

#vmworld #CNA3124BE


deep dive: vmware kubernetes engine-k8s as a service on ... · deep dive: vmware cloud pks k8s as a...

Documents