puppet camp seattle 2014: docker and puppet: 1+1=3
DESCRIPTION
Jerome Petazzoni, DockerTRANSCRIPT
Dockerand
Puppet
1+1=3
Jérôme Petazzoni(@jpetazzo)
● Grumpy French DevOps– Go away or I will replace you
with a very small shell script
● Operated and scaled dotCloud– PAAS on EC2, with LXC, Puppet,
Python, Shell, ØMQ...
Jérôme Petazzoni(@jpetazzo)
● Runs everything in containers– VPN, firewalls
– KVM, Xorg
– Docker
– …
● Helps others to do the same– CONTAINERIZE
ALL THE THINGS!!!
What is DockerThe quick elevator pitch
Docker Engine + Docker Hub
= Docker Platform
Docker Engine
The Docker Engine
● Open Source● Written in Go● Runs containers● On any modern Linux machine
(Intel 64 bits for now)
Containers ?
Containers
● Software delivery mechanism(a bit like a package!)
● Put your application in a container,run it anywhere
● A bit like a VM, but ...
I have four words for you
● CONTAINERS boot faster(than VMs)
● CONTAINERS have less overhead(more consolidation)
● CONTAINERS bring native performance(on bare metal)
● CONTAINERS are cloud-compatible(can run in VMs)
CONTAINERSboot faster
CONTAINERShave less overhead
CONTAINERSbring native performance
CONTAINERSare cloud-compatible
Docker runs on …● Bare metal
– packages, binary, CoreOS, Project Atomic, b2d...
● Desktop VM– boot2docker
● Cloud VM (Xen, ESX, KVM, HyperV...)– ready-to-run images on most public clouds
Docker Engine recap
● Approximation:it's an hypervisor to run containers
● Approximation:containers are like VMs, but lighter
● Docker makes containers available to everybody(not just veterans from the last emacs/vim war)
Stop.Demo time.
DockerHub
Docker Hub
● Services operated by Docker Inc.● Library of ready-to-use container images● Registry for your container images
(public or private)● Automated builds
(triggered by pushes to GitHub/Bitbucket)● Free for public/open source code, $$ otherwise
Buildingcontainers
Dockerfile
FROM ubuntu:14.04MAINTAINER Docker Team <[email protected]>
RUN apt-get updateRUN apt-get install -y nginxRUN echo 'Hi, I am in your container' \ >/usr/share/nginx/html/index.html
CMD [ "nginx", "-g", "daemon off;" ]
EXPOSE 80
FROM ubuntu
RUN apt-get -y updateRUN apt-get install -y g++RUN apt-get install -y erlang-dev erlang-manpages erlang-base-hipe ...RUN apt-get install -y libmozjs185-dev libicu-dev libtool ...RUN apt-get install -y make wget
RUN wget http://.../apache-couchdb-1.3.1.tar.gz | tar -C /tmp -zxf-RUN cd /tmp/apache-couchdb-* && ./configure && make install
RUN printf "[httpd]\nport = 8101\nbind_address = 0.0.0.0" > /usr/local/etc/couchdb/local.d/docker.ini
EXPOSE 8101CMD ["/usr/local/bin/couchdb"]
docker build -t jpetazzo/couchdb .
Dockerfilesvs.
Shell scripts
Shell scripts
● OK-ish for simple stacks● Tricky to handle all possible situations
(that's why we have proper config management)
Shell scripts: the dilemma
Run from scratch every time
● Pros:– no side-effect, 100% repeatability
● Cons:– create machine each time
– provision all the things, install tons of packages...
– takes forever
– you will eventually get bored and give up
Run iteratively over and over
● Pros:– much faster
● Cons:– have to deal with leftovers of previous run
– have to make sure everything is idempotent
– quickly gets tedious
– you will eventually reinvent CM
The answer:Dockerfiles
Best of both worlds
● Build from scratch everytime(re-apply each command on top of clean build)
● Build fast(by re-using snapshots of previous runs)
● Win!
Dockerfilevs.
Configuration Management
Configuration Management:the Good
● Deals with low-level stuff● Abstracts some details (distro, sometimes OS)● Ensures convergence to a known state● Library of reusable, composable templates
Configuration Management:the Bad
● Steep learning curve● Generally requires an agent
(or something to trigger e.g. « puppet apply »)● Resource-intensive
(it's OK to run the agent on a 64 GB server,it's less OK to run 100 agents on said server)
Configuration Management
● Reusability is just as good as modules are(i.e. YMMV)
● Not as deterministic as you think● Rollbacks are harder than you think
{ 'openssl' : ensure => present }
{ 'openssl' : ensure => '1.2.3-no-poodle-pls' }
Dockerfileto the rescue
Dockerfile
● Doesn't have to deal with « low-level stuff »(hardware, drivers... handled by the host)
● Doesn't need all the goodness of CM(because it doesn't have to converge)
● Partial rebuilds are fast(layered caching rebuilds only what is needed)
● Allows inheritance and composition(FROM <mycustombase>; see also: ONBUILD)
● Easy learning curve(if you know Shell, you already know Dockerfile)
But...
● Doesn't deal with « low-level stuff »(hardware, drivers...)
● Doesn't define resource dependencies(no before/after)
● Doesn't define what runs where
Puppetto the rescue
Before/After
● Use Puppet tosetup hardware(or virtual hardware), install packages, deploy code,run services.
● Use Puppet tosetup hardware(or virtual hardware), install Docker,run containers.
● Use Dockerfilesto install packages,deploy code,run services.
Do one thing,and do it well
;
First things first
https://github.com/garethr/garethr-docker
https://forge.puppetlabs.com/garethr/docker
Installing Docker with Puppet
include 'docker'
class { 'docker': version => '1.3.1'}
Warm up our image collection
# download the registry imagedocker::image { 'postgresql':}
# don't download all ubuntu,# just '14.04'docker::image { 'ubuntu': image_tag => '14.04'}
Run containers
docker::run { 'slavedb': image => 'jpetazzo/postgresql' command => '…' ports => ['5432', '22'], links => ['masterdb:master'], use_name => true, volumes => ['/var/lib/postgresql'], volumes_from => '420fc7e8aa20', memory_limit => 100000000, # bytes username => 'postgres', hostname => 'sdb.prod.dckr.io', env => ['FUZZINESS=42', FOO=BAR', 'FOO2=BAR2'], dns => ['8.8.8.8', '8.8.4.4'], restart_service => true
}
Can I use Puppet to build Docker
container images?
YES
Should I use Puppet to build Docker
container images?
NO
OK,let's do it anyway
My other VM is a container
● write a Dockerfile to install Puppet● start tons of containers● run Puppet in them (agent, or one-shot apply)
Good if you want a mix of containers/VM/metal
But slower to deploy, and uses more resources
FROM ubuntu:12.04RUN apt-get install -qy wgetRUN mkdir /puppetWORKDIR /puppetRUN wget -q http://apt.puppetlabs.com/puppetlabs-release-precise.debRUN dpkg -i puppetlabs-release-precise.debRUN apt-get update -qRUN apt-get install -qy puppet-commonCMD puppet agent --no-daemonize --verbose
Sample Dockerfile
Lightweight, portable VMs
● Start containers instead of VMs– I can start 10 containers on this puny laptop!
– You can start those 10 containers too!(Even though you have a totally different laptop!)
– We can start those containers in the Cloud!
● Deploy sshd, syslogd, crond, etc.– You can... But do you have to?
The revolution will be containerized
● write a Dockerfile to install Puppet● … and run Puppet as part of build process● deploy fully baked, « golden » images
Faster to deploy
Easier to rollback
FROM ubuntu:12.04RUN apt-get install -qy wgetRUN mkdir /puppetWORKDIR /puppetRUN wget -q http://apt.puppetlabs.com/puppetlabs-release-precise.debRUN dpkg -i puppetlabs-release-precise.debRUN apt-get update -qRUN apt-get install -qy puppet-commonENV FACTER_HOSTNAME database42ADD ./site.pp /puppet/site.ppRUN puppet apply site.pp
Sample Dockerfile
Beyond Golden
Containers
Separation of Operational Concerns
Wat?
What does that mean?
● Don't rebuild your app to change logging, remote access, and other unrelated things
● Have different policies in prod/dev/QA/etc● Ship lighter containers
Virtual Machine deployment
● Linux base system● Libraries● Application● Logging● Backups● Metrics● ...
With configuration management
node www {
include common
include web
include logstash
include backup
include graphite
}
Problems
● Conflicts between two components– e.g. logging and metrics use different Java versions
● Software certified for different distro– e.g. something wants RHEL 6.4 but you run Ubuntu
● Migration from one component to another– example: from syslog to splunk
Container deployment
● Linux base system● Docker● Application container● Logging container● Backups container● Metrics container● ...
More about that
http://blog.docker.com/2014/06/why-you-dont-need-to-run-sshd-in-docker/
http://www.slideshare.net/jpetazzo/containerization-new-virtualization-docker-separation-operational-concerns
Thoughts...
What if we could...
● Run the Puppet agent outside of the container● Run a single agent for many containers● Share the cost of the agent
Thank you!
Would You Like To Know More?
● Now: ask me questions!● Next hour: ask me more questions!● Tomorrow: Docker mini-training (11am)● Run a containers BoF at LISA?● Later: www.docker.com, #docker, docker-user...