Patrick Chanezon, @chanezon, Docker Inc.

Building Distributed Systems without DockerUsing Docker Plumbing Projects

David Chung, @dchungsf, Docker Inc.Phil Estes @estep, IBM

French

Polyglot

Platforms

Software Plumber

San Francisco

Developer Relations

@chanezon

The world needstools of mass innovation

A programmable Internet would be the ultimate tool of mass innovation

A commercial product,

built ona development platform,

built oninfrastructure,

built onstandards.

Docker is building a stack to program the Internet

Docker Platform

Isolation using Linux kernel featuresnamespaces

pid mnt net uts ipc user

cgroups memory cpu blkio devices

Image layers

1.

Developer experience

1. Get out of the way

The best tools…

2. Adapt to you

3. Make thepowerful simple

Docker for Mac Docker for Windows

2.

Orchestration

Introducing the best way to orchestrate Docker: Docker.

Docker 1.12: now with orchestration built-in.

Swarm mode

Service API

Cryptographic node identity

Built-in routing mesh

Docker 1.12: now with orchestration built-in.

Using the beta? You already have 1.12 installed.

> docker swarm init> docker service create

3.

Ops experience

Deep integration with native load-balancers, templates,SSH keys, ACLs, scaling groups, firewall rules…

beta.docker.com

Docker CaaS

Goals

+ +

Agility Portability Control

BUILDDevelopment Environments

SHIPRegistry: Secure Content &

Collaboration

RUNControl Plane: Deploy,

Orchestrate, Manage, Scale

Networking Volumes MonitoringLoggingConfig MgtCI/CD

IT OperationsDevelopers IT Operations

Docker CaaS Workflow

Docker Containers as a Service platform

BUILDDeveloper Workflows

SHIPRegistry Services

RUNManagement

Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane

Docker Cloud

Docker Container Engine

Ecosystem Plugins and Integrations

Plumbing

2013

- 05

2013

- 06

2013

- 07

2013

- 08

2013

- 09

2013

- 10

2013

- 11

2013

- 12

2014

- 01

1,000,0000

2014

- 02

2014

- 03

2014

- 04

2014

- 05

2014

- 06

2014

- 07

2014

- 08

2014

- 09

2014

- 10

2014

- 11

2014

- 12

2015

- 01

2015

- 02

2015

- 03

2015

- 04

2015

- 05

2015

- 06

2015

- 07

2015

- 08

2015

- 09

2015

- 10

2015

- 11

2015

- 12

2016

- 0

1

1,000,000,000~

10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000

6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000

2013

- 05

2013

- 06

2013

- 07

2013

- 08

2013

- 09

2013

- 10

2013

- 11

2013

- 12

2014

- 01

1,000,0000

2014

- 02

2014

- 03

2014

- 04

2014

- 05

2014

- 06

2014

- 07

2014

- 08

2014

- 09

2014

- 10

2014

- 11

2014

- 12

2015

- 01

2015

- 02

2015

- 03

2015

- 04

2015

- 05

2015

- 06

2015

- 07

2015

- 08

2015

- 09

2015

- 10

2015

- 11

2015

- 12

2016

- 0

1

~

2016

- 09

1,000,000,000~

10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000

6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000

NotaryrunC •

containerd •

HyperKit , VPNKit, DataKit •

SwarmKit •

libcontainer •

libnetwork • • Docker 1.8 : Docker Content Trust

• Docker for Mac Docker for Windows

• Docker 1.12 with built-in orchestration

• Docker 0.9 : Pluggable execution

• Docker 1.7 : Multi-Host Networking

• Docker 1.11: OCI support

Notary“Let’s stop using curl|sh”

Trusted collections for any content

Transport-agnostic

Reliable updates, proof of origin, resistant to untrusted transport, survivable key compromise

Build on industry-leading standards and research

RunCThe universal container runtime

https://runc.io

containerdA daemon to control runC

built for performance and density

http://containerd.tools/

containerd

Docker 1.11

Docker for Mac architecture(simplified)

Hypervisor Framework vmnet Framework

Docker Container Engine

Hypervisor

Linux

VPN

Data Service

Interface

Client Libraries

Admin GUI

CLI

Security Sandbox

Docker for Mac internals

Unikernels

http://unikernel.org/

Hypervisor Framework vmnet Framework

Docker Container Engine

Hyperkit

Linux

VPNKit

DataKit

Client Libraries

Admin GUI

CLI

Security Sandbox

Improving Docker with unikernel tech

runC

Open Container Initiative (OCI)

An open governance

structure for creating open

industry standards: a common

container runtime and image format.

•A Linux Foundation Collaborative Project

•Free from control by any particular vendor’s specific cloud stack or ecosystem

• Includes a specification, reference runtime* and now, a specified image format

*seeded with runc + libcontainer by Docker

OCI Specs & Status> Announced June

20th, 2015> Charter signed on

December 8th, 2015

> 49 current member companies

> Both specifications nearing 1.0 release targets

https://opencontainers.orghttps://github.com/

opencontainers

> Runtime specification: Release 1.0.0-rc2 / September 2016

https://github.com/opencontainers/runtime-spec/releases/tag/v1.0.0-rc2

1. Very close to an official 1.0 release of the runtime spec 2. Includes required core for Linux, Windows, and Solaris> Image format specification: Release 0.5.0 /

September 2016https://github.com/opencontainers/image-spec/releases/tag/v0.5.0

1. Seeded with Docker registry v2.2 specification 2. v1.0.0-rc1 release being voted/approved on mailing list

Introduction to `runc`> runc is a client wrapper around

libcontainer> Libcontainer is the OS level interface for

containersOther platforms and architectures can implement the libcontainer API via their

own primitives/system-level container concepts

$ docker run -it --read-only -v /host:/hostpath alpine sh/#

{ "ociVersion": "0.6.0-dev", "platform": { "os": "linux", "arch": "amd64" }, "process": { "terminal": true, "args": [ "sh" ], "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/bin”

config.json

● CloudFoundry Garden OCI implementation

● https://github.com/cloudfoundry-incubator/guardian

● Uses runc as a backend for container execution

● Docker 1.11 (and above)● Switched from direct libcontainer API

linkage to calling runc as container executor

● Uses containerd as a gRPC daemon to disconnect Docker daemon (API/mgmt) from container execution (allows daemon restart in future without container runtime impact)

runc in the “Wild”

runv - Hyper.sh; small & lightweight hypervisor wraps contained processrunz - Solaris zones implementation

> Ports/Implementations:

runc: An open innovation platform for containers

Implement low-level container featuresOperating system level features should be defined in the OCI runtime

specificationNew capabilities (PID cgroup controls, checkpoint/restore, seccomp)

implemented in runC

INTEREST

OCI compliance/pluggable execution engineImplement a OS/environment for containers via an OCI spec compliant binaryExamples: runz (Solaris zones), runv (hypervisor-based), Intel Clear Containers

Iterative container configuration test/debugSimple variant of “Docker-like” containers with less friction for quick modificationsLow bar for dependencies: single binary + physical rootfs bundle + JSON config

INTEREST

INTEREST

How does Docker use runc?

Docker engine

containerd

gRPC

ctr-shim ctr-shim

runc runc

https://github.com/docker/docker

https://github.com/docker/containerd

https://github.com/opencontainers/runc

Docker client/APIHTTP/REST

OCI & runc Futures

● Entry point for OS-level container technology implementations and added enhancements• Recent examples: seccomp, user namespaces,

checkpoint/restore• Many smaller examples (lots of changes required for fully

unprivileged containers)● More users and contributed implementations

(for runtime and image)● What will you do with runc?

@estesp

github.com/estesp

estesp@gmail.com

https://integratedcode.us

IRC: estesp

Phil Estes, IBMDEMO$ runc run alpine# /

InfraKit

Problem:

Managing Docker on different infrastructure isdifficult and not portable.

Consistent User Experience

How do we handle updates to a cluster??

Docker for AWS

EBS ELB

Container Engine

Storage plugin

Infrastructure Management

Network plugin Orchestration

IAM

CloudFormation

EC2VPC

Admin interface

Linux

User Applications / Services

Docker for AWS

EBS ELB

Container Engine

Storage plugin

InfraKit

Network plugin Orchestration

IAM

CloudFormation

EC2VPC

Admin interface

Linux

User Applications / Services

InfraKit

A toolkit for building declarative, self-healing infrastructure.

Declarative

• JSON configuration for desired infrastructure state:• Specification of instances — vm image, instance type, etc.• Group properties — size, logical identifiers, etc.

• Design patterns encourage • encapsulation• composition

• Config is input to all operations — system figures out what to do

Self-healing

• Composed of a set of active components / processes that• monitor infrastructure state• detect state divergence• take actions

• Continuous monitoring and reconciliation — always on

• No downtime — rolling update

Toolkit

• Primitives for managing collections of resources• create, scale, destroy• rolling update

• Abstractions & Developer SPI• Group - manages collection of resources• Instance - describes the physical resource• Flavor - extra semantics for handling instances

• A collection of executable, active components — plugins• Initially, Go daemons in the toolkit• Soon, easy management via Docker Plugins (runc)

Architecture

Instance Plugin

• Spec: specification / model of an instance (e.g. vagrant, EC2):• Logical ID, Init, Tags, and attachment• Platform-specific properties

• Methods:• /Instance.Validate• /Instance.Provision• /Instance.Destroy• /Instance.DescribeInstances

• Examples: instance plugins for EC2, Azure VM, Vagrant, …

Flavor Plugin

• Gives more context about the group members:• Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’)• Application-specific notions of ‘health’

Is the node not only present but also joined a swarm?• Methods:

• /Flavor.Validate• /Flavor.Prepare• /Flavor.Healthy

• Examples: flavor for Zookeeper members, Docker swarm nodes

Group Plugin

• Main entry point for user interaction:• Create, describe update, update, destroy• Config JSON is always the input

• Composed of Instance and Flavor — mix and match to manage cattle (fungible) or pets (special)

• Methods:• /Group.Watch• /Group.Unwatch• /Group.Inspect

• /Group.DescribeUpdate• /Group.Update• /Group.StopUpdate

• /Group.Destroy

ConfigurationExample config file (zk.conf): Group configuration = Instance + Flavor

{ "Properties": {

/* raw configuration */

}}

{ "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } }}

Operations

• Make sure the plugins are running:• infrakit/group &; infrakit/zookeeper &; infrakit/vagrant &;

• “Watch” the group starts management:• infrakit/cli group watch zk.conf

• Update the config, e.g. change size or add IP address• Describe changes before committing —

infrakit/cli group describe zk.conf• Begin update —

infrakit/cli group update zk.conf

Demo

Today

• InfraKit is just getting started… only primitives for working with groups like clusters of hosts

• But we have big plans• Improve group management strategies• More resource types — networking, load

balancers, storage…

• A cohesive framework for active management of infrastructure — physical, virtual, or containers

Get Involved

• Help define and implement new and interesting plugins• Instance plugins for different infrastructure providers • Flavor plugins for systems like etcd or mysql clusters• Group controller plugins — metrics-driven auto scaling

and more

• Help define interfaces and implement new infrastructure resource types — load balancers, networks and storage volume provisioners

More Info

• Github: https://github.com/docker/infrakit

• A quick tutorial: https://github.com/docker/infrakit/blob/master/docs/tutorial.md

Booth D38 @ LinuxCon + ContainerCon

Tues Oct 4th• Build Distributed Systems without Docker, using Docker Plumbing Projects - Patrick Chanezon, David Chung and Captain Phil

Estes• Getting Started with Docker Services - Mike Goelzer• Swarmkit: Docker’s Simplified Model for Complex Orchestration - Stephen Day• User Namespace and Seccomp Support in Docker Engine - Paul Novarese• Build Efficient Parallel Testing Systems with Docker - Docker Captain Laura Frank

Wed Oct 5th• How Secure is your Container? A Docker Engine Security Update - Phil Estes• Docker Orchestration: Beyond the Basics - Aaron Lehmann• When the Going gets Tough, get TUF Going - Riyaz Faizullabhoy and Lily Guo

Thurs Oct 6th• Orchestrating Linux Containers while Tolerating Failures - Drew Erny• Unikernels: When you Should and When you Shouldn’t - Amir Chaudhry• Berlin Docker Meetup

Friday Oct 7th• Tutorial: Comparing Container Orchestration Tools - Neependra Khare• Tutorial: Orchestrate Containers in Production at Scale with Docker Swarm - Jerome Petazzoni

THANK YOU