deploying kubernetes in the enterprise with cisco aci · container orchestration basic features
TRANSCRIPT
![Page 1: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/1.jpg)
![Page 2: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/2.jpg)
Deploying Kubernetes in the Enterprise with Cisco ACI
Camillo Rossi – TME INSBU
BRKACI-2505
![Page 3: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/3.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to communicate with the speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKACI-2505
![Page 4: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/4.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
At the end of the session, the participants should be able to:
• Have a general understanding of containers
• Have a general understanding of Kubernetes
• Understand how ACI and Kubernetes integration is deployed
Initial assumption:
• The audience already has a good knowledge of ACI main concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)
4BRKACI-2505
![Page 5: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/5.jpg)
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
![Page 6: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/6.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Kubernetes Integration – Value proposition
• Allow containers direct access to the ACI policy model, so that they can participate as first-class citizens within an ACI fabric
• Allow seamless integration of containers, VMs, and physical devices on an ACI fabric
• Support native policy semantics, so that a container application that is specified using Kubernetes NetworkPolicy will work correctly out of the box
• I.E. The same config works on Google Cloud, AWS and ACI
• Leverage fabric resources and Opflex to assist accelerating Kubernetes service load balancing
6BRKACI-2505
![Page 7: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/7.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Kubernetes Integration – Value proposition (cont.)
• Security Policies within and between pods are defined using Kubernetes Network Security API. Implemented on OVS via Host Protection Profiles, visible by APIC
• ACI admin can (optionally) define EPGs and contracts that are exposed to Kubernetes users
• EPG is selected based on annotation, which can also be used to dynamically define new EPGs
7BRKACI-2505
![Page 8: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/8.jpg)
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
![Page 9: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/9.jpg)
Linux fundamentalsUnderstanding Containers Origins
![Page 10: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/10.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Lightweight Process Virtualization is not new
• Create multiple views of the root filesystem to isolate applications and processes
• Lightweight Process Virtualization is not a new concept:
• Solaris Zones
• BSD jails
• Linux chroot
• AIX WPARs (Workload Partitions)
10BRKACI-2505
![Page 11: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/11.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is new?
• Linux Kernel 3.8 (February 2013) added support for namespaces and cgroups
• Namespaces are like chroot for processes, but also applied to network, UTS (Unix Timesharing), mount, IPC and users (UIDs).
• Example: Network namespaces enable the creation of multiple, isolated routing tables that operate independently. Multiple hostnames could also be used: • #ip netns add myns1 -> Create a new Namespace “myns1”
BRKACI-2505 11
![Page 12: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/12.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is new?
• Cgroups provide resource management capabilities
• Processes can be grouped into user-defined group of tasks, for optimized system resource usage
• Cgroups move resource allocation from the process level to the application level by grouping and labeling processes into hierarchies
• Resource allocation includes CPU time, block IO, RAM and network bandwidth
BRKACI-2505 14
![Page 13: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/13.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is new?
• Cgroups architecture:
CPU Network Memory Storage I/O
Cgroup1
Cgroup2
BRKACI-2505 15
![Page 14: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/14.jpg)
What are containers?
![Page 15: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/15.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is a container?
• A container is a binary executable, packaged with dependencies and intended for execution in a private namespace with optional resource constraints.
• This provides the containers multiple isolated operating system environments with their own file system, network, process and block I/O space on the same host
BRKACI-2505 17
![Page 16: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/16.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Compute Virtualization != Containers
Host OS
Hypervisor
App App App
Server Hardware
Host OS
Container Engine (Linux)
Guest
OS
Guest
OS
Guest
OS
VMs
Containers
Bins/
Libs
Bins/
Libs
Bins/
Libs
Server Hardware
App App App
Bins/
Libs
Bins/
Libs
Bins/
Libs
Virtualization Containers
BRKACI-2505 18
![Page 17: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/17.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19BRKACI-2505
Compute Virtualization & Containers
• They provide a way to abstract resources
• They define logical boundaries to the resources they consume
• They enable multiple OS instances to run on the same host
• They share the resources of the host
Similarities
![Page 18: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/18.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20BRKACI-2505
Compute Virtualization & Containers
• Containers can only run the same OS as the host
• Containers share the same kernel as the host
• Containers are faster to provision and boot
• Containers have lower overhead as there is no need for the hypervisor layer
Differences
![Page 19: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/19.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers Current Challenges
• Containers images management
• Orchestration of containers across multiple hosts
• Lack of standards
• Integration with virtualization and cloud tools
• Networking management
21BRKACI-2505
Addressed by ACI
![Page 20: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/20.jpg)
Why Containers?
![Page 21: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/21.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Microservice
Single Purpose
Stateless
Independently Scalable
Automated
Service
Autonomous
Loosely-coupled
Function
Single Action
Event Sourced
Ephemeral
Application Architectural Evolution
23BRKACI-2505
f()
![Page 22: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/22.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
An effective platform for micro-services
• Containers are ideal candidates to run micro-services:
• Micro-services define stateless, loosely coupled application components communicating over API’s, running in different runtime environments.
• Containers meet new application requirements as they provide:• Density
• Speed
• Portability
• Low overhead management
24BRKACI-2505
![Page 23: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/23.jpg)
Containers runtimes
![Page 24: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/24.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers runtimes
• A container runtime enables users to make effective use of containerization mechanisms by providing APIs and tooling that abstract the low level technical details
• LXC - Open Source Solutions (OSS)
• Docker – OSS and commercial
• Rkt – part of CoreOS, OSS and commercial
• VMware Integrated Container (aka Project Bonneville) - Proprietary
• RunC - OSS
• Garden – part of Pivotal Cloud Foundry, OSS and commercial
BRKACI-2505
![Page 25: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/25.jpg)
Docker
![Page 26: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/26.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker provides an integrated technology suite that enables development and IT operations teams to build, ship, and run distributed applications anywhere.
30BRKACI-2505
![Page 27: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/27.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
A Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries –anything that can be installed on a server.
This guarantees that the software will always run the same, regardless of its environment.
31BRKACI-2505
![Page 28: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/28.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker consists of two main components:
33BRKACI-2505
Docker Hub – SaaS component for managing
and sharing containers.
Docker Engine – the
actual app running
on the host.
Containers
![Page 29: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/29.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dockerfile and Registry
• A Dockerfile is simply a text file containing instructions on how to build a Docker image
• It can add components on top of an existing image
• Images are available online on the Docker hub repository
• Local, private registry can be created.
• A registry is an instance of the registry container image
34BRKACI-2505
![Page 30: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/30.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dockerfile and Registry
• A Dockerfile is simply a text file containing instructions on how to build a Docker image
• It can add components on top of an existing image
• Images are available online on the Docker hub repository
• Local, private registry can be created.
• A registry is an instance of the registry container image
35BRKACI-2505
![Page 31: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/31.jpg)
More about Docker Images…
![Page 32: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/32.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• A Docker image is made up of filesystems layered over each other.
37BRKACI-2505
![Page 33: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/33.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• The storage driver is responsible for presenting these layers as a single, unified file system.
38BRKACI-2505
![Page 34: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/34.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• When you start a container, Docker creates an empty, read-write layer on top of the stack – all changes are made in this layer.
39BRKACI-2505
![Page 35: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/35.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker Images
• Docker uses “copy-on-write” container layers.
• If a file needs to be modified, it is copied into the read-write layer first.
40BRKACI-2505
![Page 36: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/36.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• This means that multiple containers can share a single copy of the image.
41BRKACI-2505
![Page 37: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/37.jpg)
Docker Networking
![Page 38: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/38.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 1 - None
• Doesn’t create any network interface for the container
43BRKACI-2505
![Page 39: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/39.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 2 - Bridge Mode
• Default mode where Docker attaches containers to Docker0 bridge
• Containers in the same host can talk to each other
• Containers on different host can’t talk to each other (or anything else) easily
44BRKACI-2505
Host-2
Container 3
docker0 Bridge
172.17.42.1
Eth0172.17.0.12
veth994786d
Container 4
Eth0172.17.0.13
vethab4e22e
Host-1
Container 1
docker0 Bridge
172.17.42.1
Eth0172.17.0.12
veth774786d
Container 2
Eth0172.17.0.13
vethde4e22e
Same IPs for the containers is not a typo
![Page 40: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/40.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 2 - Bridge Mode
• It’s possible to expose a container externally by publishing a port (NAT)
45BRKACI-2505
Host
Container 1
NGINX
docker0 Bridge
![Page 41: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/41.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 3 - Host Mode
• Connect containers to Host network stack
• All the network interfaces defined on the host will be available to the container (every container will have the same IP address as the host)
46BRKACI-2505
Host-1
Container 1
Eth0 (host interface)
192.168.0.2
Eth0192.168.0.2
Container 2
Eth0192.168.0.2
![Page 42: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/42.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Option 4 - Mapped Container Mode
• Container is mapped to another container network stack
• Filesystem, processes and other resources are kept separate
• They share network resources (IP, interfaces)
47BRKACI-2505
Host-1
Container 1
docker0 Bridge
172.17.42.1
Eth0172.17.0.12
veth774786d
Container 2
Eth0172.17.0.12
![Page 43: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/43.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
All these Docker Networking options are complex
• Iptables rules must be created manually to allow/NAT traffic to containers ports…
• Containers on different hosts can’t communicated to each other even if they are in the same L2 domain
• Needs to manually manage port-mappings
• Prone to errors
48BRKACI-2505
![Page 44: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/44.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Docker network driver plugins
• Network plugins can be used to extended Docker networking support to a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different.
49BRKACI-2505
![Page 45: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/45.jpg)
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
![Page 46: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/46.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
In a multi-host environment, containers need to
• Have network reachability
• Be fault-tolerant
• Easily scalable
• Use resources optimally
• Can discover other containers/application automatically
• Communicate with each other
• Can be update/rollback without any downtime
• Expose services in an easy and reliable way
53BRKACI-2505
![Page 47: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/47.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container Orchestration basic features
• Bring multiple hosts together and make them part of a cluster
• Schedule containers to run on different hosts
• Help containers running on one host reach out to containers running on other hosts in the cluster
• Bind containers and storage
• Bind containers of similar type to a higher-level construct, like services, so we don't have to deal with individual containers
• Keep resource usage in-check, and optimize it when necessary
• Allow secure access to applications running inside containers.
BRKACI-2505 54
![Page 48: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/48.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Containers Orchestrators
• Docker Swarm is a Container Orchestrator provided by Docker, Inc. It is part of Docker Engine.
• Kubernetes started by Google, now part of the Cloud Native Computing Foundation project.
• Mesos Marathon is one of the frameworks to run containers at scale on Apache Mesos.
• Amazon EC2 Container Service (ECS) is a hosted service provided by AWS to run Docker containers at scale on its infrastructure.
• Hashicorp Nomad is the Container Orchestrator provided by HashiCorp.
BRKACI-2505 55
![Page 49: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/49.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes
• Kubernetes is an open source Container Orchestration system for automating deployment, scaling and management of containerized applications.
• It was inspired by the Google Borg System and with its v1.0 release in July 2015, Google donated it to the Cloud Native Computing Foundation (CNCF).
• Generally, Kubernetes has new releases every three months. The current stable version is 1.9 (as of Jan 2018).
BRKACI-2505 56
![Page 50: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/50.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes & Docker
• Kubernetes uses Docker to execute/run the containers
• Kubernetes adds, on top of Docker, all the intelligence and features of an orchestrator
BRKACI-2505 57
![Page 51: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/51.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Features
• Automatic binpackingKubernetes automatically schedules the containers based on resource usage and constraints, without sacrificing availability.
• Self-healingKubernetes automatically replaces and reschedules the containers from failed nodes. It also kills and restarts containers which do not respond to health checks, based on existing rules/policy.
• Horizontal scalingKubernetes can automatically scale applications based on resource usage like CPU and memory. It also supports dynamic scaling based on customer metrics
BRKACI-2505 58
![Page 52: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/52.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Features (cont.)
• Service discovery and Load balancingKubernetes groups sets of containers and refers to them via a DNS name. This DNS name is also called a Kubernetes service. Kubernetes can discover these services automatically, and load-balance requests between containers of a given service.
• Automated rollouts and rollbacksKubernetes can roll out and roll back new versions/configurations of an application, without introducing any downtime.
• Secrets and configuration managementKubernetes can manage secrets and configuration details for an application without re-building the respective images. With secrets, we can share confidential information to our application without exposing it to the stack configuration, like on GitHub.
BRKACI-2505 59
![Page 53: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/53.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Features (cont.)
• Storage orchestrationWith Kubernetes and its plugins, we can automatically mount local, external, and storage solutions to the containers in a seamless manner, based on Software Defined Storage (SDS).
• Batch executionBesides long running jobs, Kubernetes also supports batch execution.
BRKACI-2505 60
![Page 54: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/54.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Architecture
• At a very high level, Kubernetes has the following main components:
• One or more Master Nodes
• One or more Worker Nodes
• Distributed key-value store, like etcd.
BRKACI-2505 61
![Page 55: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/55.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Components – Master Node
• The Master Node is responsible for managing the Kubernetes cluster. Master node access methods are CLI, GUI or APIs.
• For fault tolerance, there can be more than one Master Node.
• To manage the cluster state, Kubernetes uses etcd, and all Master Nodes connect to it. etcd is a distributed key-value store. The key-value store can be part of the Master Node. It can also be configured externally, in which case, the Master Nodes connect to it.
BRKACI-2505 63
![Page 56: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/56.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Components – Worker Node
• A Worker Node is a machine (VM, physical server, etc.) which runs the containers using pods and is controlled by the Master Node.
• pods are scheduled on the Worker Nodes
BRKACI-2505 64
![Page 57: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/57.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes - pod
• A pod is the scheduling unit in Kubernetes. It is a logical collection of one or more containers which are always scheduled together.
• The set of containers composed together in a pod share an IP.
[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system
NAME READY STATUS RESTARTS AGE
aci-containers-controller-1201600828-qsw5g 1/1 Running 1 69d
aci-containers-host-lt9kl 3/3 Running 0 72d
aci-containers-host-xnwkr 3/3 Running 0 58d
aci-containers-openvswitch-0rjbw 1/1 Running 0 58d
aci-containers-openvswitch-7j1h5 1/1 Running 0 72d
BRKACI-2505 65
![Page 58: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/58.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Deployment
• Deployments are a collection of pods providing the same service
• You describe the desired state in a Deployment object, and the Deployment controller will change the actual state to the desired state at a controlled rate for you
• For example you can create a deployment that declare you need to have 2 copies of your front-end pod.
[root@k8s-01-p1 ~]# kubectl get deployment --namespace=kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
aci-containers-controller 1 1 1 1 72d
BRKACI-2505 66
![Page 59: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/59.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Services
• A service tells the rest of the Kubernetes environment (including other pods and Deployments) what services your application provides.
• While pods come and go, the service IP addresses and ports remain the same.
• Kubernetes automatically load balance the load across the replicas in the deployment that you expose through a Service
• Other applications can find your service through Kurbernetes service discovery.
• Every time a service is create a DNS entry is added to kube-dns
[root@k8s-01-p1 ~]# kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 11.96.0.10 <none> 53/UDP,53/TCP 72d
BRKACI-2505 67
![Page 60: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/60.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – External Services
• If there are external IPs that route to one or more cluster nodes, Kubernetes services can be exposed on those external IPs.
• Traffic that ingresses into the cluster with the external IP (as destination IP), on the service port, will be routed to one of the service endpoints.
• External IPs are not managed by Kubernetes and are the responsibility of the cluster administrator.
[root@k8s-01-p1 ~]# kubectl get svc front-end --namespace=guest-book
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
front-end 11.96.0.33 11.3.0.2 80:30002/TCP 3m
BRKACI-2505 68
![Page 61: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/61.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Ingress
• An Ingress is a collection of rules that allow inbound connections to reach the cluster services.
• It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting, and more
• Think of NGINX
[root@k8s-01-p1 ~]# kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
test-ingress * 80 7s
BRKACI-2505 69
![Page 62: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/62.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes - Labels
• Kubernetes uses labels as “nametags” to identify things.
• Can be used to indicate roles, stability, or other important attributes.
• You can query anything in Kubernetes via a label.
• i.e. Return all the pod that are running “PreProduction” workload
[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system -l component=kube-apiserver
NAME READY STATUS RESTARTS AGE
kube-apiserver-k8s-01-p1 1/1 Running 0 72d
BRKACI-2505 70
![Page 63: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/63.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes - Annotations
• Similar to labels but are NOT used to identify and select object
• Used in ACI, yes soon we will be speaking about ACI and Kubernetes
[root@k8s-01-p1 ~]# kubectl describe node k8s-01-p1 | more
Name: k8s-01-p1
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=k8s-01-p1
node-role.kubernetes.io/master=
Annotations: node.alpha.kubernetes.io/ttl=0
opflex.cisco.com/pod-network-ranges={"V4":[{"start":"11.2.0.130","end":"11.2.1.1"}]}
opflex.cisco.com/service-endpoint={"mac":"66:85:9a:e9:ef:2f","ipv4":"11.5.0.3"}
volumes.kubernetes.io/controller-managed-attach-detach=true
BRKACI-2505 71
![Page 64: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/64.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Namespace
• Groups everything together:
• Pod
• Deployment
• Volumes
• Services
• Etc…
BRKACI-2505 72
![Page 65: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/65.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
All Together: A K8S Cluster
Container
Application
pod1
Deployment1
Container
Application
pod2
Container
Application
pod[n]
Node1
Node[N]
Node2Service
1.1.1.1:80
Namespace
A node can be part of
Several Namespaces
BRKACI-2505 73
![Page 66: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/66.jpg)
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Agenda
![Page 67: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/67.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Application Centric InfrastructureAny Application – Any hypervisor
• Policy consistency provides for
containers running reliably and
securely
• Build to and support open systems
and standards
• Common pervasive gateway and
policy based routing provide optimal
network connectivity
• Ease of deploying, scaling and
managing
BRKACI-2505 75
![Page 68: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/68.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco ACI and Container Integration
Node
OpFlex OVS
ACI and Containers
Unified networking: Containers, VMs, and
bare-metal
Micro-services load balancing integrated in
fabric for HA / performance
Secure multi-tenancy and seamless
integration of Kubernetes network policies
and ACI policies
Visibility: Live statistics in APIC per
container and health metricsNode
OpFlex OVS
BRKACI-2505 76
![Page 69: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/69.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
`
1
2 Deploy/Scale Clusters
Build containers
WEB APP WEB APP DB
Server 1 Server 2
Opflex/OVS
Host level Policy
Enforcement
ACI Fabric
Define BDs, Context and
AP
Fabric Bring Up
2
1
Get VLAN Pools Allocated
For Each EPG3
Full Infrastructure Visibility,
Telemetry4
Annotate policy3EPG
Developer Network Administrator
Infosec
Define Container
Network Policy
Infrastructure Policy
Enforcement
1
77BRKACI-2505
ACI Network Plugin for KubernetesNative Security Policy Support
![Page 70: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/70.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI VMM Domain for Kubernetes
78BRKACI-2505
Node
OpFlex OVS
Kubernetes
ACI Policies
Technical DescriptionNetwork Policy
Node
OpFlex OVS
• Network policies of Kubernetes supported using standard
upstream format but enforced through OpFlex / OVS using
APIC Host Protection Profiles
• Kubernetes app configurations can be moved without
modification to/from ACI and non-ACI environments
• Embedded fabric and virtual switch load balancing
• PBR in fabric for external service load balancing
• OVS used for internal service load balancing
• VMM Domain for Kubernetes
• Stats per namespace, deployment, service, pod
• Physical to container correlation
![Page 71: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/71.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI CNI Plugin Components
• aci-containers-controller
• Handle IPAM
• Management of endpoint state
• Policy Mapping (annotations)
• Controls Load Balancing
• Pushes configurations into the APIC
79BRKACI-2505
![Page 72: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/72.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI CNI Plugin Components
• aci-containers-host is a DaemonSet composed of 3 containers:
• mcast-daemon: • Handles Broadcast, unknown unicast and multicast replication
• aci-containers-host:• Endpoint metadata
• Pod IP Address management
• Container Interface Configuration
• opflex-agent:• Support for Stateful Security Groups
• Manage configuration of OVS
• Render policy to openflow rules to program OVS.
• Handles loadbalanced services (connection tracking, natting, etc…)
80BRKACI-2505
![Page 73: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/73.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI CNI Plugin Components
• aci-containers-openvswitch
• Bridge traffic from containers to physical interfaces
• Enforce policies
81BRKACI-2505
![Page 74: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/74.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82BRKACI-2505
Flexible Definition of EPG Boundaries
Cluster Isolation Namespace Isolation Deployment Isolation
pod
pod
pod
pod
pod
pod
• Single EPG for entire cluster,
default behavior
• No need for any internal contracts
• Each namespace is mapped to its
own EPG
• Contracts for inter-namespace traffic
• Each deployment mapped to an EPG
• Contracts tightly control service traffic
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
Contra
ct
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
pod
Contra
ct
Contract
Contract
Contra
ct
EPG
Network
Policy
![Page 75: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/75.jpg)
ACI Basic Configuration
![Page 76: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/76.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
acc-provision
• ACI Container Controller Provision:
• Takes a YAML file containing the parameters of your configuration
• Generates and pushes most of the ACI config
• Generates Kubernetes ACI CNI containers configuration
acc-provision --flavor=kubernetes-1.7 -a -u admin -p pass –c config.yml –o cni_config.yml
84BRKACI-2505
Used to select if we are deploying
kubernetes 1.6, 1.7 or OpenShift 3.6
APIC user and
passwordConfiguration file
Output file for ACI CNI
config
![Page 77: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/77.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
acc-provision – configuration file (1)
aci_config:
system_id: KubeSpray # Tenant Name and Controller Domain Name
apic_hosts: # List of APIC hosts to connect for APIC API
- 10.67.185.102
vmm_domain: # Kubernetes VMM domain configuration
encap_type: vxlan # Encap mode: vxlan or vlan
mcast_range: # mcast range for BUM replication
start: 225.22.1.1
end: 225.22.255.255
mcast_fabric: 225.1.2.4
nested_inside: # (OPTIONAL) If running k8s node as VMs specify the VMM Type and Name.
type: vmware # Only vmware for now, ports groups created automatically with system_id name
name: ACI
# The following resources must already exist on the APIC,
# they are used, but not created by the provisioning tool.
aep: ACI_AttEntityP # The AEP for ports/VPCs used by this cluster
vrf: # The VRF can be placed in the same Tenant or in Common.
name: vrf1
tenant: KubeSpray # This can be the system-id or common
l3out:
name: l3out # Used to provision external IPs
external_networks:
- default_extepg # Default Ext EPG, used for PBR redirection
85BRKACI-2505
![Page 78: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/78.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
acc-provision – configuration file (2)
#
# Networks used by Kubernetes
#
net_config:
node_subnet: 10.32.0.1/16 # Subnet to use for nodes
pod_subnet: 10.33.0.1/16 # Subnet to use for Kubernetes Pods
extern_dynamic: 10.34.0.1/24 # Subnet to use for dynamic external IPs
extern_static: 10.35.0.1/24 # Subnet to use for static external IPs
node_svc_subnet: 10.36.0.1/24 # Subnet to use for service graph
kubeapi_vlan: 4011 # The VLAN used by for nodes to node API communications
service_vlan: 4013 # The VLAN used by LoadBalancer services
infra_vlan: 3456 # The ACI infra VLAN used to establish the OpFlex tunnel with the leaf
#
# Configuration for container registry
# Update if a custom container registry has been setup
#
registry:
image_prefix: noiro # DO NOT CHANGE
86BRKACI-2505
![Page 79: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/79.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87BRKACI-2505
ACI Fabric Configuration - EPGs
• kube-nodes: node API communication• Physical Domain
• kube-system: k8s components running as pods• Container Domain*
• kube-default: All user created pods will be placed in this EPG by default• Container Domain*
* Equivalent of VMM Domain for Containers
![Page 80: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/80.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88BRKACI-2505
ACI Fabric Configuration – BDs and VRF
• kube-nodes-bd:• Only used for kube-node EPG
• Maps to node_subnet
• kube-pod-bd: • Any pod will be assigned an IP from
this BD Subnet
• Used for kube-default, kube-system and any other user defined EPGs.
• Maps to pod_subnet
• KubeSpray…-service:• BD for PBR/SG services
• Created when ACI CNI plugin is deployed
![Page 81: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/81.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89BRKACI-2505
ACI Fabric Configuration – L3OUT
• User must create and configure the L3OUT
• L3OUT and Default Networks names must match the acc-provision config
![Page 82: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/82.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90BRKACI-2505
ACI Fabric Configuration – Contracts
• Minimum sets configured to ensure basic cluster functionality
• User can define additional contracts as usual
![Page 83: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/83.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91BRKACI-2505
ACI Fabric Configuration – L4-L7 Devices
• Created once the ACI CNI plugin is deployed
• Dynamically updated if nodes are added or removed from the k8s cluster
• Service Graph Template: Specify a template for PBR redirection
![Page 84: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/84.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI Fabric Configuration – Container Domain
APIC keeps inventory of pods
and their metadata (labels,
annotations), deployments,
replicasets, etc.
View pods per node, map to
encapsulation, physical point
in the fabric.
Fabric admin can search
APIC for k8s nodes, masters,
pods, services …
BRKACI-2505 92
![Page 85: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/85.jpg)
Demo 1 Deploying an Application
![Page 86: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/86.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Spin up a guestbook application
• The guestbook application uses Redis to store its data. It writes its data to a Redis master instance and reads data from multiple Redis slave instances.
• The code can be found at: https://kubernetes.io/docs/tutorials/stateless-application/guestbook/
94BRKACI-2505
For your
reference
![Page 87: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/87.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Spin up a guestbook application
• Create a namespace for our application
cisco@k8s-01:~/demo/guestbook1$ kubectl create namespace guestbook
• Deploy all the componentscisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook apply -f complete.yaml
deployment "frontend" created
service "frontend" created
deployment "redis-master" created
service "redis-master" created
deployment "redis-slave" created
service "redis-slave" created
• Check POD status cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get pod -o wide
READY STATUS RESTARTS AGE IP NODE
frontend-1768566195-mj43h 1/1 Running 0 2m 10.33.1.11 k8s-02
frontend-1768566195-tpw75 1/1 Running 0 2m 10.33.0.153 k8s-03
frontend-1768566195-vljrh 1/1 Running 0 2m 10.33.0.155 k8s-03
redis-master-2365125485-8hg60 1/1 Running 0 2m 10.33.0.152 k8s-03
redis-slave-3837281623-p4fs7 1/1 Running 0 2m 10.33.1.12 k8s-02
redis-slave-3837281623-qw894 1/1 Running 0 2m 10.33.0.154 k8s-03
Note: All the commands are executed from the Kubernetes master node95BRKACI-2505
For your
reference
![Page 88: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/88.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Check APIC Controller Domain
96BRKACI-2505
APIC has complete visibility into
k8s objects
Visibility on where this specific
pod is running
For your
reference
![Page 89: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/89.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Check APIC EPG
97BRKACI-2505
POD Name
For your
reference
![Page 90: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/90.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 1 – Cluster Services
• By default every POD is exposed only to the k8s cluster via a Service IP. You can imagine this as a Virtual IP of a load balancer.
• With the ACI CNI plugin the LoadBalancing for the internal cluster services is performed by OVS cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend 10.37.0.124 <nodes> 80:32677/TCP 25m
redis-master 10.37.0.162 <none> 6379/TCP 25m
redis-slave 10.37.0.136 <none> 6379/TCP 25m
• Try to access the service from one of the nodescisco@k8s-01:~/demo/guestbook1$ curl 10.37.0.124
<html ng-app="redis">
<head>
<title>Guestbook</title>
!SNIP!
98BRKACI-2505
For your
reference
![Page 91: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/91.jpg)
Demo 2 Placing PODs/Namespaces in EPGs
![Page 92: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/92.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 2 - APIC Steps
• Create an EPG under your application
• BD = your pod BD
• VMM Domain = Your Kubernetes Domain
• Every POD in an EPG needs to be able to communicate with:
• kube-system for cluster wide DNS resolution
• kube-node for health monitoring probes
• Top Tip: Use EPG contract masters and inherit contracts from kube-default!
100BRKACI-2505
For your
reference
![Page 93: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/93.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo 2 - acikubectl
• Utility to manage and troubleshoot the k8s cluster
• Can be used to annotate Namespeces or Deployments with the Tenant/App/EPG names
cisco@k8s-01:~/demo/guestbook1$ acikubectl set default-eg namespace guestbook -t KubeSpray -a kubernetes -g
guestbook
Setting default endpoint group:
Endpoint Group:
Tenant: KubeSpray
App profile: kubernetes
Endpoint group: guestbook
BRKACI-2505
For your
reference
![Page 94: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/94.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Check under your EPG
• All your PODs should now have moved from kube-default to guestbook
102BRKACI-2505
For your
reference
![Page 95: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/95.jpg)
Exposing Services
![Page 96: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/96.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
The extern_dynamic subnet
• Defined in acc-provision configuration file
• An IP address will be automatically selected from this subnet to expose your service outside of the k8s cluster/fabric
• Expose the service as “LoadBalancer” (as per kubernetes standard)
• The extern_dynamic subnet is not associated to a BD: You need to configure your external router with static routes toward your L3OUT for this subnet
104BRKACI-2505
cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get svc frontend
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend 10.37.0.124 10.34.0.5 80:32677/TCP 5h
extern_dynamic
![Page 97: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/97.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR
105BRKACI-2505
* defined in the acc-provision config file
Every time a service is exposed the ACI CNI controller will deploy:
• An External EPG with a /32 match for the Service IP
• A new contract between the svc_ExtEPG and the default_ExtEPG*
• A Service Graph with PBR redirection containing every node where an exposed POD is running
L3Out
default_ExtEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
![Page 98: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/98.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
106BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
1. Client send a request to 10.34.0.2, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG
Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
![Page 99: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/99.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
107BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
Node1
Node2
NodeN
RTRClient
Cons
Prov
1. Client send a request to 10.34.0.5, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG
2. ACI does a routing lookup for 10.34.0.5, IP does not exist in the fabric, we should route it out however
LPM places it in the Svc_x_ExtEPG
3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to one of the nodes
SIP DIP
192.168.1.100 10.34.0.5
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
![Page 100: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/100.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
108BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
Node1
Node2
NodeN
RTRClient
Cons
Prov
1. Client send a request to 10.34.0.2, ACI performs Longest Prefix Match (LPM) on the SIP and classify the traffic in the default_extEPG
2. ACI does a routing lookup for 10.34.0.2, IP does not exist in the fabric, we should route it out however
LPM places it in the Svc_x_ExtEPG
3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to one of the nodes
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
SIP DIP
192.168.1.100 10.34.0.5
![Page 101: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/101.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
109BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
4. The K8S node is not expecting any traffic directed to the external service IP so OVS will perform NAT as required
5. If there are multiple POD on a single node OVS will perform a second stage LB to distribute the load between Pods running on the same node
Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
SIP DIP
192.168.1.100 PodX IP
DIP
192.168.1.100 10.34.0.5
![Page 102: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/102.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Graphs and PBR – Packet walk
110BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Svc_x_ExtEPG
10.34.0.5/32
Contract PBR Service Graph
RTRClient
Cons
Prov
4. PodX replies to the client
5. OVS restore the original external Service IP
6. PBR redirection is not triggered since the source EPG is the Shadow EPG of the PBR node
7. Traffic is routed back to the client (and is permitted by the contract)
DIP SIP
192.168.1.100 10.34.0.5Node1
Node2
NodeN
Pod1
Pod3
Pod5
Pod2
Pod4
NodeN
OVS
OVS
OVS
DIP SIP
192.168.1.100 PodX IP
![Page 103: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/103.jpg)
Demo 3 Exposing Services
![Page 104: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/104.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Exposing a service
• Simply choose the LoadBalancer ”type” in the service definition
• The ACI CNI plug in will:
• Automatically pick a free IP from the extern_dynamic subnet
• Create the ExtEPG
• Create contracts
• Create PBR redirection rules
• Deploy the service graph
112BRKACI-2505
![Page 105: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/105.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Scalability
• Currently the scalability of exposing external service with PBR is limited by the number of external EPGs per L3OUT.
• ACI 3.1 supports 250 external EPGs per L3 OUT per leaf *
• This is a soft limit and will increase with time
• But we want more! So?
113BRKACI-2505
*For details check:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/3-x/verified_scalabilty/b_Verified_Scalability_3_1_1x_and_13_1_1x.html
![Page 106: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/106.jpg)
Scaling External Services with Ingress
![Page 107: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/107.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes – Ingress
• Composed of two parts:
• Ingress Resources: collection of rules that defines how inbound connections can reach the internal cluster services.
• Ingress controller: responsible for fulfilling the Ingress, usually with a virtual loadbalancer (nginx, ha-proxy)
• Ingress controller can be shared between multiple namespaces
• It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting etc…
• Bonus Feature: Easy integration with DNS: configure a wildcard DNS record (*.cisco.com) pointing to the IP of the ingress controller
115BRKACI-2505
![Page 108: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/108.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ingress Cont 1
Ingress Cont 2
Ingress Cont N
Namespace
ingress
Kubernetes – Ingress
116BRKACI-2505
Pod1
Pod2
Namespace
ciscolive
Namespace
cisco
Service
Pod1
Pod2
Service
Ingress
Controller
Ingress Resource
(I am www.cisco.com)
Ingress Resource
(I am www.ciscolive.com)
Clientwww.cisco.com
www.ciscolive.com
![Page 109: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/109.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI and Kubernetes Ingress
• Expose the Ingress Controller via Service Graph with PBR
• A single Service Graph/ExtEPG can now host as many services as we want
• Ingress Controller can be scaled (creating multiple instances) as needed
• If you create a dedicated EPG for ingress you need the following contracts:
• All the contracts used in kube-default (remember contract inheritance)
• Consume: Kube-API, Ingress need to be able to speak with the Kube API server
• Consume: any required ports between Ingress Controller and the service you wan to expose
117BRKACI-2505
![Page 110: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/110.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI and Kubernetes Ingress
118BRKACI-2505
L3Out
default_extEpg
0.0.0.0/0
Ingress_ExtEPG
10.34.0.6/32
Contract PBR Service Graph
RTR
Client
Cons
Prov
Ingress Cont 1
Ingress Cont 2
Ingress Cont N
Pod1
Pod2
Namespace
ingress
Namespace
ciscolive
Namespace
cisco
Service
Pod1
Pod2
Service
Ingress
Controller
Ingress Resource
(I am www.cisco.com)
Ingress Resource
(I am www.ciscolive.com)
![Page 111: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/111.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI and Kubernetes External Services - Summary
• Two options (can be used at the same time even for the same service)
• Exposing services via ingress
• Exposing up to 250 services directly with Service Graph with PBR
119BRKACI-2505
![Page 112: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/112.jpg)
Demo 4 Exposing Services with Ingress
![Page 113: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/113.jpg)
Kubernetes Cluster Node Failure
![Page 114: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/114.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kubernetes Cluster Node Failure Detection
• Kubernetes Monitors by default all the node in the clusters
• Depending on the configuration, node failure detection and container restart can take from ~50s to 5min. This will depend on your specific configuration.
• Once a node is detected as NotReady (failed) the aci-container-controller will update the ACI configuration as required i.e. a failed node will be removed from the PBR redirection policy
122BRKACI-2505
![Page 115: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/115.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 123BRKACI-2505
ACI CNI redundancy during node failure
• DataPlane of the CNI Plugin
• Start and Stop with the Node
• If isolated from the network they will try to reconnect to the leaf
aci-containers-host & aci-containers-openvswitch
![Page 116: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/116.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 124BRKACI-2505
ACI CNI redundancy during node failure
• Stateless
• Does not sit in the data-path
• In case of failure k8s will restart it on a different node
aci-containers-controller (acc)
![Page 117: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/117.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 125BRKACI-2505
ACI CNI redundancy during node failure – Corner case
• Node connects to OOB and ACI Fabric
• K8S Cluster communications are happening over the ACI Fabric
• acc communicates with APIC via OOB
aci-containers-controller (acc)
Node1
OOB
aci-containers-controller1
ACI
Fabric
Node2
![Page 118: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/118.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 126BRKACI-2505
ACI CNI redundancy during node failure – Corner case
• Node 1 losses connectivity with the ACI Fabric (interface down)
• Kubernetes master will detect node1 as lost and restart acc on Node2
• The old instance of acc1 is still running and will keep injecting the old config, overwriting the configuration changes pushed by acc2
• When designing your network ensure that acc communication with the APIC goes trough the fabric
aci-containers-controller
Node1
OOB
aci-containers-controller1
ACI
Fabric
Node2 aci-containers-controller2
![Page 119: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/119.jpg)
Container to Non-Container Communications
![Page 120: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/120.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Container to Non-Container Communications
• In production environments is preferred, for example, to run services like high performance databases as VMs or Bare Metal Servers
• This calls for the ability to easily provide communication between Kubernetes POD and VMs/Bare Metal
• Simply deploy a contract between your EPGs, ACI will do the rest!
• This works for any VMM domain and Physical Domains, for example you can have a Container Domain using VXLAN speaking with a Microsoft SCVMM Domain using VLAN.
128BRKACI-2505
![Page 121: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/121.jpg)
How can I test this?
![Page 122: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/122.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
LABACI-2010 ACI Runs Everything
• WISP Lab
• Come to Build a WordPress application running on:
• Kubernetes
• Hyper-V
• Bare Metal
• ESXi
130BRKACI-2505
S_HAProxy
Percona 1Cluster
BareMetal
Percona 3Cluster
Persona 2 Cluster
L3OUT
NGINXWordPress01
NGINXWordPress02
SCVMM
ESXiBareMetal
Kubernetes
Client
Tenant-Common
Tenant-Common
Tenant-CL18-PODX
![Page 123: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/123.jpg)
How can build my own lab?
![Page 124: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/124.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
kubespray_aci!
• Fork of kubespray
• Modified to deploy an high available cluster using ACI CNI plugin
• Open Source (not supported by TAC/Cisco etc…)
• Optionally can clone VM templates and configure everything providing a 1-Click deployment solution for your lab
• https://github.com/camrossi/kubespray_aci
132BRKACI-2505
Not officially
supported
Yes it is me… Did I
mentioned is not
officially supported?
![Page 125: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/125.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 133BRKACI-2505
Contiv 1.0 and Contiv-ACIContiv 1.0 in ACI Mode
Docker 1.12+
Support
OpenShift 3.4+
Integration
Physical Domain /
REST API
Contiv-ACI
Open Source at github.com/contiv
Kubernetes 1.6, 1.7
VMM Domain /
OpFlex
OpenShift Integration
Contiv Solutions
Support
![Page 126: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/126.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 134BRKACI-2505
Platform Summary for ACIPlatform Contiv-ACI (VMM Domain) Contiv 1.0 (Physical Domain)
GA ACI 3.0 Not recommended
GA ACI 3.1 Supported.
Version 3.4 / RHEL 7.x
Beta ACI 3.1 Not supported
Future Supported.
Engine 1.12.x / Swarm 1.2.5 / Centos 7.x
1.1.6 + Engine 1.12.x / RHEL 7.x
2.1.x + Engine 17.03-ee / RHEL 7.x
Future TBD
Future Future
Kubernetes
Nomad
![Page 127: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/127.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to communicate with the speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKACI-2505
![Page 128: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/128.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Please complete your Online Session Evaluations after each session
• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt
• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.
Complete Your Online Session Evaluation
![Page 129: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/129.jpg)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
137BRKACI-2505
![Page 130: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/130.jpg)
Thank you
![Page 131: Deploying Kubernetes in the Enterprise with Cisco ACI · Container Orchestration basic features](https://reader030.vdocuments.site/reader030/viewer/2022020100/5b94d9e509d3f2a65f8de5da/html5/thumbnails/131.jpg)