docker and puppet for continuous integration
TRANSCRIPT
About me• 15 years “in the trenches cubicles”
• Developer of RTC (VoIP, IM, WebRTC) solutions
• Founder of RTCSoft in 2015
@giavac
https://github.com/giavac
Docker usage scenarios• Run services
• Packaging/Deployment mechanism
• Fast prototyping
• Continuous Integration/Delivery
Continuous Integration 1/2
• A team of developers work on the same project
• Each developer commits several times per day
• Each commit is potentially verified (Build, Deploy, Test)
• The project is always kept in a stable/releasable state
Continuous Integration 2/2
• Developers work on personal/bug/feature (short-lived) branches
• These branches are merged into the “release” branch for the release
Image source: www.atlassian.com
CI components
Image source: myself (This is why it sucks)
Server-side apps for Linux
• Apps are built, packaged, deployed and tested
• Unit testing during the builds
• Component/Integration/Load testing after test deployments
• Artefacts are uploaded into file servers or deb/yum repos
Old school: “Build and run”Build the app on the runtime machine
• PROBLEMS:
• Repeat the build on each host
• Unnecessary pollution of the runtime host
• Maintenance nightmare: the easiest upgrade is problematic
• Potential exposure of source code to external attacks
Using a “Build machine”Dedicate a machine to the various builds
• PROs:
• Keep the runtime environment reasonably clean
• Source code is protected inside the private network
• CONs:
• You’re reducing a multi-dimensional problem into a bi-dimensional one
• You’ll pay the price one day!
• Widely adopted, Open Source CI tool
• Core + many plugins
• Integrates well with git and Docker
• Can manage multiple “slaves”
https://jenkins-ci.org/
Jenkins and Master worker• “Cron jobs manager with a GUI”
• What’s a Jenkins job?
• The simplest case: Jenkins runs the jobs inside its own host
• “Master” worker, no slaves involved
• Still you need to manage Jenkins’ configuration…
Building machineSAssign a VM per building scenario (app/OS)
• A step in the right direction
• CONs:
• Hard to maintain (see Configuration Management later)
• VM resources allocated even when the builds are not running
• Explosion of number of VM types to use (OS/versions matrix)
• VMs are “heavy”
Jenkins and Slaves
• Configure a number of “slave workers”
• Assign a slave to a job
• But you need to maintain all those slaves…
Configuration Management
You need something better than that.
Configuration Management intro• Define programmatically the configuration of a machine
• Many tools available: Puppet, Ansible, Chef, Salt, etc.
• Make configuration predictable AND fix divergence automatically
• Track changes during time (it’s all text)
• Define a formal language across developers, sysadmin, Ops
• Puppet defines the final state of a host (What, not How)
• Has its own declarative syntax
• Written in Ruby
• Designed for Master-Slave interaction, can be used Standalone
• Idempotent
https://puppetlabs.com/
Configuration Management
You really, really need it. Even if you like 3-ways diffs and hate the term “idempotence”.
Example of Puppet code package { 'nginx': ensure => present, } -> file { '/etc/nginx/nginx.conf': ensure => file, owner => 'root', group => 'root', mode => '0644', content => template('mymodule/nginx.conf.erb'), notify => Service['nginx'], } -> file { '/etc/nginx/conf.d/global.conf': ensure => file, owner => 'root', group => 'root', mode => '0644', content => template('mymodule/nginx_global.conf.erb'), notify => Service['nginx'], }
service { 'nginx': ensure => running, enable => true, }
Back to Jenkins and multiple Slaves
• We saw the approach with multiple slaves, with VMs as slave
• Functionally valid
• Cumbersome to maintain
• What if we could have “on demand”, lightweight workers? e.g. Docker containers?
• A new form of virtualisation (w/ real HW performance)
• A delivery mechanism
• Many “images” freely available
• “Here’s the ingredients and recipe” VS “Here’s the cake”
https://www.docker.com/
Virtual Machines vs Docker
Source: https://www.docker.com/what-docker
Docker workflow• Build an image (Dockerfile + base images)
• docker build -t gvacca/nginx .
• Store image in a registry (Public, e.g. Dockerhub or private)
• docker push gvacca/nginx
• Pull image on a target host
• docker pull gvacca/nginx
• Run container on a target host
• docker run -it gvacca/nginx
Dockerfile - exampleFROM ubuntu:latest
MAINTAINER Giacomo Vacca <[email protected]>
RUN apt-get update
# Install nginx RUN apt-get -y -q install nginx
# Prepare target dir for website (Must match global.conf) RUN mkdir -p /var/www/html
# Configure nginx COPY nginx/nginx.conf /etc/nginx/nginx.conf COPY nginx/global.conf /etc/nginx/conf.d/global.conf
EXPOSE 8080
CMD [ "/usr/sbin/nginx" ]
https://github.com/giavac/docker-experiments/tree/master/simple_nginx
Build a Docker imagegvacca@dockerubuntu:~simple_nginx$ sudo docker build -t gvacca/simple-nginx . Sending build context to Docker daemon 7.168 kB Step 1 : FROM ubuntu:14.04 ---> 1d073211c498 … Step 2 : MAINTAINER Giacomo Vacca <[email protected]> ---> Running in 10a8334becfd ---> 559310abf4fb Removing intermediate container 10a8334becfd Step 3 : RUN apt-get update ---> Running in 873b09debab0 … Step 8 : EXPOSE 8080 ---> Running in a27f73413c02 ---> f5ef8527f2a3 Removing intermediate container a27f73413c02 Step 9 : CMD /usr/sbin/nginx ---> Running in d99de19cc782 ---> df76d20cd00f Removing intermediate container d99de19cc782 Successfully built df76d20cd00f
Run the container!gvacca@dockerubuntu:simple_nginx$ sudo docker run -d --name nginx -p 8080:8080 -v $PWD/website:/var/www/html/website gvacca/simple-nginx add8d371f82f615ebbd121cea804511dcdafac893b14325366744797424fa44c
gvacca@dockerubuntu:simple_nginx$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES add8d371f82f gvacca/simple-nginx "/usr/sbin/nginx" 11 seconds ago Up 9 seconds 0.0.0.0:8080->8080/tcp nginx
gvacca@dockerubuntu:simple_nginx$ ps aux |grep docker root 792 0.0 4.7 693136 23740 ? Ssl Nov04 11:20 /usr/bin/docker daemon root 1977 0.0 2.6 152692 13172 ? Sl 11:26 0:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.2 -container-port 8080
Intermediate topics• Volumes
• Network (was ‘link’ before 1.9)
• Debugging (‘inspect’)
• ARG (after 1.9)
• Docker on OSX/Windows (Docker Machine)
• Interacting with Kernel modules
Docker orchestration• Docker Composer
• Docker Swarm
• Google’s Kubernetes
• Apache Mesos
• Puppet & others
Let’s start exploring interaction opportunities between the tools
seen so far…
Puppet to manage Jenkins• https://forge.puppetlabs.com/rtyler/jenkins
• Manage Jenkins as a service
• Install plugins automatically, etc
• Or build your own module!
• Install .war
• Configure Jenkins (config.xml)
• Configure jobs ($JENKINS_PATH/jobs/JOB/config.xml)
Docker as Slave for Jenkins• Associate specific Docker images to specific Jenkins build jobs
• Keep builds clean & reproducible (“Non-event releases”)
• Run slave containers on demand - stop them when not needed.
• Jenkins Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
• Other Cloud methods, e.g. Mesos
Jenkins to build Docker images• A Jenkins job:
• Checks out a Dockerfile
• Builds the image
• Upload the new image in a repo
• May re-use images for new builds (careful about integrity)
Puppet to manage Docker containers• Module for docker: https://forge.puppetlabs.com/garethr/docker
• Installs Docker and required dependencies
• Launches the Docker daemon
• Sets DNS, users and other configuration items
• Pull images from selected repos
• Run containers (image + command)
Run Jenkins inside Docker• Manage Jenkins with a dedicated image + volume with data
• From the official Docker registry: https://hub.docker.com/_/jenkins/
docker run -d -p 8080:8080 -p 50000:50000 -v ~:/var/jenkins_home jenkins
• TCP 8080: Web UI
• TCP 50000: interface to connect slaves
Docker inside DockerBut you need the privileged mode
Conclusion• There are many opportunities to automate Continuous Integration
• CI for server-side apps poses specific challenges
• You can’t ignore Configuration Management
• Docker represents a huge opportunity
• You need to find your own use case and solution
APPENDIX - Puppet vs Docker
• Some stop using Puppet for Configuration Management once they move their apps inside containers
• A Dockerfile defines “how” to build an image
• A Dockerfile must be distribution-specific (yum or apt?)
• Puppet is OS-agnostic: the differences are hidden inside modules
Questions and AnswersThanks
Recommended Books• “The Docker book”, J. Turnbull, http://www.amazon.co.uk/Docker-
Book-Containerization-new-virtualization-ebook/dp/B00LRROTI4
• “Continuous Delivery”, J. Humble, http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
• “Pro Puppet”, J. Turnbull, http://www.amazon.co.uk/Pro-Puppet-Second-Professional-Apress/dp/1430260408
See also…
http://www.slideshare.net/GiacomoVacca/when-voip-meets-web-the-revolution-of-webrtc