Containers Infrastructure for Advanced Management

Federico Simoncelli – Principal Software EngineerRed Hat

July 2015

AgendaRunning containers on an infrastructure is a “solved problem”.

Now you have to run them smoothly.

• Deployment• Metrics Technologies• Analyzing and Correlating Metrics• Smart Analysis – Fleecing• Cross-Provider Relationships• Traceability• Advanced Management

Requirements• Docker

automates the deployment of applications inside software containers

• Kubernetesmanages a cluster of Linux containers as a single system

• OpenShiftbuild, deploy, and manage your applications with Docker and Kubernetes

• Project Atomiclightweight operating system (rpm-ostree) assembled out of RPM content

• Foremancomplete lifecycle management tool for physical and virtual servers

Deployment - Choices• Deploying Atomic

• http://www.projectatomic.io (Fedora and CentOS based images)• Bootable, immutable, versioned filesystem, updates delivered in large sets• Everything that is not in the base system runs as a container

• Deploying a Traditional Distribution• More flexible in nodes content• Matrix of components is less tested

• Deployment on IaaS (OpenStack, oVirt)• Secure – Improved isolation of tenancy• “Elastic” on demand

• Deployment on Bare Metal• Automated with Foreman or Satellite

Deployment - Technologies• Kubernetes project uses Salt

• Requires a salt-master and salt-minion running on the nodes• Mostly for GCE (and Vagrant)

• OpenShift project uses Ansible• https://github.com/openshift/openshift-ansible• Supports AWS GCE libvirt OpenStack Vagrant

• OpenStack• Heat - Kubernetes and OpenShift templates can be found on github repositories• Magnum - Container orchestration engines as first class resources in OpenStack

• Self-Registration of nodes• Enabled by default - preferred pattern used by most distributions• Can be disabled for strict security policies

Metrics Collection• Containerized solution• Collecting Metrics

• Heapster• Kubelet (cAdvisor)

• Storing Metrics• InfluxDB• Hawkular

• Metrics Analisys• Metrics per Container / Pod / Service / Replica• Highly coupled with the metrics database

Kubernetes Node 1 Kubernetes Node N

Heapster Metrics DB

Kubernetes Master Metrics Analysis

IaaS Host YIaaS Host X

Kubernetes Node 1 Kubernetes Node N

Metrics Cross-Linking• IaaS metrics collection

• OpenStack Ceilometer• oVirt Data Warehouse

• Metrics collection and analysis• Nodes, Instances and Hosts

Heapster

IaaS Instance 1 IaaS Instance N

IaaS Metrics

Host load spike

Instance load spike

Metrics – Hawkular• Open Source monitoring solution - http://www.hawkular.org• Components can be re-used in applications outside of Hawkular• Capable of receiving monitoring data from various existing sources• Scalable and high performance metric storage based on Cassandra• On-premise or in the cloud (works on OpenShift out of the box)• Other features and work in progress

• Alerting• Data Visualization (in progress)• Analytics• Push notification (in progress)

Fleecing – Images Inspection• Case 1 – Strict policy, allow only specific known images

• Pre-compute images content• Case 2 – No policy, images are downloaded from any registry

• Images discovery• Docker signed images• Extract images (tricky)• Inspect images (e.g. OpenSCAP)• Cross-link with packages and errata database• Store/cache content and analysis (retention after image disappeared)

Fleecing – Images Inspection• Use the same nodes of the cluster infrastructure• Dockerized solution - runs everywhere (Atomic)• Should not execute anything from inside the image (best practice)• docker-fleece – https://github.com/simon3z/docker-fleece

• Creates a stopped container from image (fast)• Uses CopyFromContainer to extract files (linear)• Docker Graph Driver independent

• docker-mount – https://github.com/rhvgoyal/docker-mount• Creates a stopped container from image (fast)• Mounts externally the Container mountpoint (fast)• Tightly coupled with the Docker Graph Driver (device-mapper, overlayfs, etc.)

Fleecing – Docker Fleece Example• How to extract image content:

$ docker images

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

docker.io/fedora 22 ded7cd95e059 6 weeks ago 186.5 MB

docker.io/fsimonce/docker-fleece latest 847c8293c5b4 7 weeks ago 217.9 MB

$ docker run -ti --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock \

-v /tmp/docker-fleece:/tmp/docker-fleece fsimonce/docker-fleece --image=fedora:22 --path=/tmp/docker-fleece

(3.38 seconds)

$ du -sh /tmp/docker-fleece

194M /tmp/docker-fleece

Fleecing – Container Inspection• Inspect the content of a running container• Analyze the “drift” of a container from its original image

• were the packages/binaries tampered with?• Fleecing a running container requires snapshots and merge capabilities

• Freeze container processes• Flush cache• Snapshot and clone container• Unfreeze container processes• Analyze cloned container• Merge back container snapshot

Kubernetes - Traceability• Report Docker Container ID in Container Status

• Allows to peek in Docker (or other runtimes) low level information

• Report Docker Image ID in Container Status• Allows to reliably identify the specific images in use

• Add Node System Info• Allows to identify reboots and version of the critical components (Kernel, Docker, etc.)

• Add Instance ID and Provider ID to Nodes• Allows to identify the instances of the Kubernetes nodes

• Improved Logging• Allows to analyze important events in the cluster and infrastructure

ManageIQ• Open Source Cloud Management Platform - http://manageiq.org• Self-Service Provisioning and Chargeback• Places Virtual Workloads according to your business priorities

• Balance your costs, performance, security, and reliability across cloud platforms

• Supports Infrastructure (oVirt, VMware) and Cloud (OpenStack) Providers• Smart Analysis of Hosts, Virtual Machines and Instances

• Drift of the instances from initial image (detect new applications, new users)• Supports definition of Compliance Policies

ManageIQ - Inventory Full Picture• Inventory collection at all layers for infrastructures, clouds and containers

• Containers, Instances, Hosts, Storage and Network

Service Pods Container Image

NodesCluster

InstanceRegion

Host

Storage Network

ManageIQ - Inventory Relationship

ManageIQ - Inventory Relationship

ManageIQ - Inventory Relationship

ManageIQ - Capacity and Utilization• Metrics correlations, and in the future cross-provider correlations

ManageIQ - Dashboard and Topology

ManageIQ - Dashboard and Topology

ManageIQ - Stay in Touch• Community http://talk.manageiq.org• Code https://github.com/ManageIQ/manageiq #providers/containers• Documentation http://manageiq.org/documentation• Social:

• Twitter @manageiq #manageiq

• Contact Me• Federico Simoncelli - fsimonce@redhat.com• Twitter @simon3z