puppet camp seattle 2014: docker and puppet: 1+1=3

Dockerand

Puppet

1+1=3

Jérôme Petazzoni(@jpetazzo)

● Grumpy French DevOps– Go away or I will replace you

with a very small shell script

● Operated and scaled dotCloud– PAAS on EC2, with LXC, Puppet,

Python, Shell, ØMQ...

Jérôme Petazzoni(@jpetazzo)

● Runs everything in containers– VPN, firewalls

– KVM, Xorg

– Docker

– …

● Helps others to do the same– CONTAINERIZE

ALL THE THINGS!!!

What is DockerThe quick elevator pitch

Docker Engine + Docker Hub

= Docker Platform

Docker Engine

The Docker Engine

● Open Source● Written in Go● Runs containers● On any modern Linux machine

(Intel 64 bits for now)

Containers ?

Containers

● Software delivery mechanism(a bit like a package!)

● Put your application in a container,run it anywhere

● A bit like a VM, but ...

I have four words for you

● CONTAINERS boot faster(than VMs)

● CONTAINERS have less overhead(more consolidation)

● CONTAINERS bring native performance(on bare metal)

● CONTAINERS are cloud-compatible(can run in VMs)

CONTAINERSboot faster

CONTAINERShave less overhead

CONTAINERSbring native performance

CONTAINERSare cloud-compatible

Docker runs on …● Bare metal

– packages, binary, CoreOS, Project Atomic, b2d...

● Desktop VM– boot2docker

● Cloud VM (Xen, ESX, KVM, HyperV...)– ready-to-run images on most public clouds

Docker Engine recap

● Approximation:it's an hypervisor to run containers

● Approximation:containers are like VMs, but lighter

● Docker makes containers available to everybody(not just veterans from the last emacs/vim war)

Stop.Demo time.

DockerHub

Docker Hub

● Services operated by Docker Inc.● Library of ready-to-use container images● Registry for your container images

(public or private)● Automated builds

(triggered by pushes to GitHub/Bitbucket)● Free for public/open source code, $$ otherwise

Buildingcontainers

Dockerfile

FROM ubuntu:14.04MAINTAINER Docker Team <education@docker.com>

RUN apt-get updateRUN apt-get install -y nginxRUN echo 'Hi, I am in your container' \ >/usr/share/nginx/html/index.html

CMD [ "nginx", "-g", "daemon off;" ]

EXPOSE 80

FROM ubuntu

RUN apt-get -y updateRUN apt-get install -y g++RUN apt-get install -y erlang-dev erlang-manpages erlang-base-hipe ...RUN apt-get install -y libmozjs185-dev libicu-dev libtool ...RUN apt-get install -y make wget

RUN wget http://.../apache-couchdb-1.3.1.tar.gz | tar -C /tmp -zxf-RUN cd /tmp/apache-couchdb-* && ./configure && make install

RUN printf "[httpd]\nport = 8101\nbind_address = 0.0.0.0" > /usr/local/etc/couchdb/local.d/docker.ini

EXPOSE 8101CMD ["/usr/local/bin/couchdb"]

docker build -t jpetazzo/couchdb .

Dockerfilesvs.

Shell scripts

Shell scripts

● OK-ish for simple stacks● Tricky to handle all possible situations

(that's why we have proper config management)

Shell scripts: the dilemma

Run from scratch every time

● Pros:– no side-effect, 100% repeatability

● Cons:– create machine each time

– provision all the things, install tons of packages...

– takes forever

– you will eventually get bored and give up

Run iteratively over and over

● Pros:– much faster

● Cons:– have to deal with leftovers of previous run

– have to make sure everything is idempotent

– quickly gets tedious

– you will eventually reinvent CM

The answer:Dockerfiles

Best of both worlds

● Build from scratch everytime(re-apply each command on top of clean build)

● Build fast(by re-using snapshots of previous runs)

● Win!

Dockerfilevs.

Configuration Management

Configuration Management:the Good

● Deals with low-level stuff● Abstracts some details (distro, sometimes OS)● Ensures convergence to a known state● Library of reusable, composable templates

Configuration Management:the Bad

● Steep learning curve● Generally requires an agent

(or something to trigger e.g. « puppet apply »)● Resource-intensive

(it's OK to run the agent on a 64 GB server,it's less OK to run 100 agents on said server)

Configuration Management

● Reusability is just as good as modules are(i.e. YMMV)

● Not as deterministic as you think● Rollbacks are harder than you think

{ 'openssl' : ensure => present }

{ 'openssl' : ensure => '1.2.3-no-poodle-pls' }

Dockerfileto the rescue

Dockerfile

● Doesn't have to deal with « low-level stuff »(hardware, drivers... handled by the host)

● Doesn't need all the goodness of CM(because it doesn't have to converge)

● Partial rebuilds are fast(layered caching rebuilds only what is needed)

● Allows inheritance and composition(FROM <mycustombase>; see also: ONBUILD)

● Easy learning curve(if you know Shell, you already know Dockerfile)

But...

● Doesn't deal with « low-level stuff »(hardware, drivers...)

● Doesn't define resource dependencies(no before/after)

● Doesn't define what runs where

Puppetto the rescue

Before/After

● Use Puppet tosetup hardware(or virtual hardware), install packages, deploy code,run services.

● Use Puppet tosetup hardware(or virtual hardware), install Docker,run containers.

● Use Dockerfilesto install packages,deploy code,run services.

Do one thing,and do it well

;

First things first

https://github.com/garethr/garethr-docker

https://forge.puppetlabs.com/garethr/docker

Installing Docker with Puppet

include 'docker'

class { 'docker': version => '1.3.1'}

Warm up our image collection

# download the registry imagedocker::image { 'postgresql':}

# don't download all ubuntu,# just '14.04'docker::image { 'ubuntu': image_tag => '14.04'}

Run containers

docker::run { 'slavedb': image => 'jpetazzo/postgresql' command => '…' ports => ['5432', '22'], links => ['masterdb:master'], use_name => true, volumes => ['/var/lib/postgresql'], volumes_from => '420fc7e8aa20', memory_limit => 100000000, # bytes username => 'postgres', hostname => 'sdb.prod.dckr.io', env => ['FUZZINESS=42', FOO=BAR', 'FOO2=BAR2'], dns => ['8.8.8.8', '8.8.4.4'], restart_service => true

}

Can I use Puppet to build Docker

container images?

YES

Should I use Puppet to build Docker

container images?

NO

OK,let's do it anyway

My other VM is a container

● write a Dockerfile to install Puppet● start tons of containers● run Puppet in them (agent, or one-shot apply)

Good if you want a mix of containers/VM/metal

But slower to deploy, and uses more resources

FROM ubuntu:12.04RUN apt-get install -qy wgetRUN mkdir /puppetWORKDIR /puppetRUN wget -q http://apt.puppetlabs.com/puppetlabs-release-precise.debRUN dpkg -i puppetlabs-release-precise.debRUN apt-get update -qRUN apt-get install -qy puppet-commonCMD puppet agent --no-daemonize --verbose

Sample Dockerfile

Lightweight, portable VMs

● Start containers instead of VMs– I can start 10 containers on this puny laptop!

– You can start those 10 containers too!(Even though you have a totally different laptop!)

– We can start those containers in the Cloud!

● Deploy sshd, syslogd, crond, etc.– You can... But do you have to?

The revolution will be containerized

● write a Dockerfile to install Puppet● … and run Puppet as part of build process● deploy fully baked, « golden » images

Faster to deploy

Easier to rollback

FROM ubuntu:12.04RUN apt-get install -qy wgetRUN mkdir /puppetWORKDIR /puppetRUN wget -q http://apt.puppetlabs.com/puppetlabs-release-precise.debRUN dpkg -i puppetlabs-release-precise.debRUN apt-get update -qRUN apt-get install -qy puppet-commonENV FACTER_HOSTNAME database42ADD ./site.pp /puppet/site.ppRUN puppet apply site.pp

Sample Dockerfile

Beyond Golden

Containers

Separation of Operational Concerns

Wat?

What does that mean?

● Don't rebuild your app to change logging, remote access, and other unrelated things

● Have different policies in prod/dev/QA/etc● Ship lighter containers

Virtual Machine deployment

● Linux base system● Libraries● Application● Logging● Backups● Metrics● ...

With configuration management

node www {

include common

include web

include logstash

include backup

include graphite

}

Problems

● Conflicts between two components– e.g. logging and metrics use different Java versions

● Software certified for different distro– e.g. something wants RHEL 6.4 but you run Ubuntu

● Migration from one component to another– example: from syslog to splunk

Container deployment

● Linux base system● Docker● Application container● Logging container● Backups container● Metrics container● ...

More about that

http://blog.docker.com/2014/06/why-you-dont-need-to-run-sshd-in-docker/

http://www.slideshare.net/jpetazzo/containerization-new-virtualization-docker-separation-operational-concerns

Thoughts...

What if we could...

● Run the Puppet agent outside of the container● Run a single agent for many containers● Share the cost of the agent

Thank you!

Would You Like To Know More?

● Now: ask me questions!● Next hour: ask me more questions!● Tomorrow: Docker mini-training (11am)● Run a containers BoF at LISA?● Later: www.docker.com, #docker, docker-user...