devoxx 2016 - docker nuts and bolts
TRANSCRIPT
Patrick Chanezon@chanezon, Docker Inc.
Docker Nuts and Bolts
French
Polyglot
Platforms
Software Plumber
San Francisco
Developer Relations
@chanezon
1995 2015
PublicHybridPrivate
Ops Devops Developers
Linux Container Ecosystem
flockerglusterfs
weavecalicomidokuracisconuage
Cloud
OS
Plugins
Orchestration
Agility
Agile methodologies (circa 1999)
Low MTBIAMSH
MTBIAMSH (Mean Time Between Idea And Making Stuff Happen)
Agility == $$
Devops
25
Mainframe
Client-Server
26
27
Web
28
Cloud - Devops
Devops• Cultural movement • Inspired by agile methods• People, Processes & Tools• Continuous delivery• Infrastructure as code• Cross silo collaboration• Small iterations• Feedback loop, measurement
Image from Patrick Deboishttp://www.slideshare.net/jedi4ever/devops-the-war-is-over-if-you-want-ithttp://www.slideshare.net/jedi4ever/devopsdays-downundervfinal
Devops: singing Kumbaya?
28
http://highscalability.com/blog/2013/11/19/we-finally-cracked-the-10k-problem-this-time-for-managing-se.html
Server/Sysadmin
1999: 5(Windows) - 50 (Linux)2015: 10k-20kx2000
28
https://blog.docker.com/2014/12/dockercon-europe-keynote-continuous-delivery-in-the-enterprise-by-henk-kolk-ing/Henk Kolk, ING, DockerCon EU 2014
People, Processes, Products
deployment time: 9 months -> 15 min1500 deployments/week
Docker Platform
The world needstools of mass innovation
A programmable Internet would be the ultimate tool of mass innovation
A commercial product,
built ona development platform,
built oninfrastructure,
built onstandards.
Docker is building a stack to program the Internet
Isolation using Linux kernel featuresnamespaces pid mnt net uts ipc user
cgroups memory cpu blkio devices
Image layers
Docker for developers
https://registry.hub.docker.com/_/java/
Using Docker to compile your jar/war
https://registry.hub.docker.com/_/maven/
docker run -it --rm \-v $PWD:/usr/src/spring-doge \-v maven:/root/.m2 \-w /usr/src/spring-doge \maven:3.3-jdk-8 \mvn package
Analyzing imagesdocker images javadocker history java:8
Build an imagedocker build -t chanezon/spring-doge .
FROM java:8MAINTAINER Patrick Chanezon <[email protected]>EXPOSE 8080COPY spring-doge/target/*.jar /usr/src/spring-doge/spring-doge.jarWORKDIR /usr/src/spring-dogeCMD java -Dserver.port=8080 -Dspring.data.mongodb.uri=$MONGODB_URI -jar spring-doge.jar
Run a containerdocker run \—env MONGODB_URI=mongodb://mongo:27017/test \-p 8090:8080 \chanezon/spring-doge
docker-compose: running multiple containers Run your stack with one command: docker-compose
up Describe your stack with one file: docker-compose.ymlversion: '2'services: web: image: chanezon/spring-doge ports: - "8080:8080" links: ["mongo"] environment: - MONGODB_URI=mongodb://mongo:27017/test mongo: image: mongo
What’s New?
1.
Developer experience
1. Get out of the way
The best tools…
2. Adapt to you
3. Make thepowerful simple
Docker for Mac Docker for Windows
2.
Orchestration
Introducing the best way to orchestrate Docker: Docker.
Docker 1.12: now with orchestration built-in.
Swarm mode
Service API
Cryptographic node identity
Built-in routing mesh
Docker 1.12: now with orchestration built-in.
Using the beta? You already have 1.12 installed.
> docker swarm init> docker service create
3.
Ops experience
Deep integration with native load-balancers, templates,SSH keys, ACLs, scaling groups, firewall rules…
beta.docker.com
Distributed Application Bundle
experimenta
l www.docker.com/dab
A portable format for multi-container applications
Docker CaaS
Goals
+ +
Agility Portability Control
Cloud Zone 1
Cloud Zone 2 Data Center
Development Center
Headquarters
Containerization: standard containers on a standardized container engine
Orchestration: build and deploy complex systems easily
Enterprise: Enable delivery and time to value across a large, complex, rapidly evolving enterprise
The Global Enterprise Supply Chain
BUILDDevelopment Environments
SHIPRegistry: Secure Content &
Collaboration
RUNControl Plane: Deploy,
Orchestrate, Manage, Scale
Networking Volumes MonitoringLoggingConfig MgtCI/CD
IT OperationsDevelopers IT Operations
Docker CaaS Workflow
Docker Containers as a Service platform
BUILDDeveloper Workflows
SHIPRegistry Services
RUNManagement
Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane
Docker Cloud
Docker Container Engine
Ecosystem Plugins and Integrations
Docker UCP 1.1 - DTR 2.0• HA• Unified Auth• Compose deployment• UI to add nodes
UCP Permission Model
Security scanning in Docker Cloud
Plumbing
2013
- 05
2013
- 06
2013
- 07
2013
- 08
2013
- 09
2013
- 10
2013
- 11
2013
- 12
2014
- 01
1,000,0000
2014
- 02
2014
- 03
2014
- 04
2014
- 05
2014
- 06
2014
- 07
2014
- 08
2014
- 09
2014
- 10
2014
- 11
2014
- 12
2015
- 01
2015
- 02
2015
- 03
2015
- 04
2015
- 05
2015
- 06
2015
- 07
2015
- 08
2015
- 09
2015
- 10
2015
- 11
2015
- 12
2016
- 0
1
1,000,000,000~
10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000
6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000
2013
- 05
2013
- 06
2013
- 07
2013
- 08
2013
- 09
2013
- 10
2013
- 11
2013
- 12
2014
- 01
1,000,0000
2014
- 02
2014
- 03
2014
- 04
2014
- 05
2014
- 06
2014
- 07
2014
- 08
2014
- 09
2014
- 10
2014
- 11
2014
- 12
2015
- 01
2015
- 02
2015
- 03
2015
- 04
2015
- 05
2015
- 06
2015
- 07
2015
- 08
2015
- 09
2015
- 10
2015
- 11
2015
- 12
2016
- 0
1
~
2016
- 09
1,000,000,000~
10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000
6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000
NotaryrunC •
containerd •
HyperKit , VPNKit, DataKit •
SwarmKit •
libcontainer •
libnetwork • • Docker 1.8 : Docker Content Trust
• Docker for Mac Docker for Windows
• Docker 1.12 with built-in orchestration
• Docker 0.9 : Pluggable execution
• Docker 1.7 : Multi-Host Networking
• Docker 1.11: OCI support
Notary“Let’s stop using curl|sh”
Trusted collections for any content
Transport-agnostic
Reliable updates, proof of origin, resistant to untrusted transport, survivable key compromise
Build on industry-leading standards and research
containerdA daemon to control runC
built for performance and density
http://containerd.tools/
containerd
Docker 1.11
Docker for Mac architecture(simplified)
Hypervisor Framework vmnet Framework
Docker Container Engine
Hypervisor
Linux
VPN
Data Service
Interface
Client Libraries
Admin GUI
CLI
Security Sandbox
Docker for Mac internals
Hypervisor Framework vmnet Framework
Docker Container Engine
Hyperkit
Linux
VPNKit
DataKit
Client Libraries
Admin GUI
CLI
Security Sandbox
Improving Docker with unikernel tech
InfraKit
Problem:
Managing Docker on different infrastructure isdifficult and not portable.
Consistent User Experience
How do we handle updates to a cluster??
Docker for AWS
EBS ELB
Container Engine
Storage plugin
Infrastructure Management
Network plugin Orchestration
IAM
CloudFormation
EC2VPC
Admin interface
Linux
User Applications / Services
Docker for AWS
EBS ELB
Container Engine
Storage plugin
InfraKit
Network plugin Orchestration
IAM
CloudFormation
EC2VPC
Admin interface
Linux
User Applications / Services
InfraKit
A toolkit for building declarative, self-healing infrastructure.
Declarative
• JSON configuration for desired infrastructure state:• Specification of instances — vm image, instance type, etc.• Group properties — size, logical identifiers, etc.
• Design patterns encourage • encapsulation• composition
• Config is input to all operations — system figures out what to do
Self-healing
• Composed of a set of active components / processes that• monitor infrastructure state• detect state divergence• take actions
• Continuous monitoring and reconciliation — always on
• No downtime — rolling update
Toolkit
• Primitives for managing collections of resources• create, scale, destroy• rolling update
• Abstractions & Developer SPI• Group - manages collection of resources• Instance - describes the physical resource• Flavor - extra semantics for handling instances
• A collection of executable, active components — plugins• Initially, Go daemons in the toolkit• Soon, easy management via Docker Plugins (runc)
Architecture
Instance Plugin
• Spec: specification / model of an instance (e.g. vagrant, EC2):• Logical ID, Init, Tags, and attachment• Platform-specific properties
• Methods:• /Instance.Validate• /Instance.Provision• /Instance.Destroy• /Instance.DescribeInstances
• Examples: instance plugins for EC2, Azure VM, Vagrant, …
Flavor Plugin
• Gives more context about the group members:• Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’)• Application-specific notions of ‘health’
Is the node not only present but also joined a swarm?• Methods:
• /Flavor.Validate• /Flavor.Prepare• /Flavor.Healthy
• Examples: flavor for Zookeeper members, Docker swarm nodes
Group Plugin
• Main entry point for user interaction:• Create, describe update, update, destroy• Config JSON is always the input
• Composed of Instance and Flavor — mix and match to manage cattle (fungible) or pets (special)
• Methods:• /Group.Watch• /Group.Unwatch• /Group.Inspect
• /Group.DescribeUpdate• /Group.Update• /Group.StopUpdate
• /Group.Destroy
ConfigurationExample config file (zk.conf): Group configuration = Instance + Flavor
{ "Properties": {
/* raw configuration */
}}
{ "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } }}
Operations
• Make sure the plugins are running:• infrakit/group &; infrakit/zookeeper &; infrakit/vagrant &;
• “Watch” the group starts management:• infrakit/cli group watch zk.conf
• Update the config, e.g. change size or add IP address• Describe changes before committing —
infrakit/cli group describe zk.conf• Begin update —
infrakit/cli group update zk.conf
Today
• InfraKit is just getting started… only primitives for working with groups like clusters of hosts
• But we have big plans• Improve group management strategies• More resource types — networking, load
balancers, storage…
• A cohesive framework for active management of infrastructure — physical, virtual, or containers
Get Involved
• Help define and implement new and interesting plugins• Instance plugins for different infrastructure providers • Flavor plugins for systems like etcd or mysql clusters• Group controller plugins — metrics-driven auto scaling
and more
• Help define interfaces and implement new infrastructure resource types — load balancers, networks and storage volume provisioners
More Info
• Github: https://github.com/docker/infrakit
• A quick tutorial: https://github.com/docker/infrakit/blob/master/docs/tutorial.md
Demo
Linux X11 Apps on Docker for Mac
https://github.com/chanezon/docker-tips/x11
• Docker Swarm, Compose and networking• docker 1.11• swarm 1.1.0• compose 1.6.0 with networking• Run/Debug with STS IDE in
a container
Spring Boot App using MongoDB
https://github.com/joshlong/spring-doge
https://github.com/chanezon/docker-tips/https://github.com/chanezon/spring-doge
Docker Java Labs
https://github.com/docker/labs/tree/master/developer-tools/
• Wildfly and Couchbase J2EE App• Debugging a Java app in Docker using Eclipse
THANK YOU