1.Continuous Deployment with JenkinsStephen ConnollyArchitect, CloudBeesTL;DR http://goo.gl/kNgyu 2012 CloudBees, Inc.All Rights Reserved

2. Who am I? One of the first non-Sun contributors toJenkins. Creator of the Weather Icons! Architect @ CloudBees Member & Maven PMC @ Apache SoftwareFoundation In my previous lives: Software Engineer @ Avaya IT Project Manager @ Elan Pharmaceuticals PhD development chemist @ Bristol Myer Squibb Freelance software developer since 19892012 CloudBees, Inc.2 All Rights Reserved 3. JenkinsWhat the feck is this Jenkins thing anyway?2012 CloudBees, Inc. All Rights Reserved 4. Introducing Jenkins What is Jenkins? A relatively recent product, but growing in popularity Currently used by*: Yahoo NASA JBoss / RedHat Amazon HP And more!*Source: Kohsuke Kawaguchi, lead developer of the Jenkins project4 5. Introducing Jenkins What makes Jenkins so great?Distributed BuildsReporting Plugins Ease of use 5 6. jenkins-ci.org OSS CI server Written in Java About 7 years old Easy to install/use Extensible via 530+ plugins Widely adopted 40K+ installations 2012 CloudBees, Inc. 6All Rights Reserved 7. The Deed Poll sed -e s/hudson/jenkins/g You might be more14000 Total created ticketsfamiliar with the IP 12000 Jenkinsencumbered old nameHudson 10000of the project.8000 The project wasrenamed to Jenkins in 6000January 2011. 4000 Nearly 100 releases of2000Jenkins since the 0 Total resolvedrebrandtickets About 8 releases ofHudson since the fork. 2012 CloudBees, Inc. 7All Rights Reserved 8. Just for Continuous Integration?Yeah! Look mate, this is a PUPPET conference not a ContinuousIntegration conference show me that youre relevant before I startlobbing soft fruits!2012 CloudBees, Inc. All Rights Reserved 9. Puppet manifests are code too TL;DRpuppet parser validate Syntax Check Use puppet parser validate, e.g. for file in $(find . -iname *.pp) do puppet parser validate--render-as s--modulepath=modules"$file" || exit 1; done2012 CloudBees, Inc.9 All Rights Reserved 10. Puppet manifests are code too Syntax Check Static Analysis Checks for anti-patterns Checks conformance to the style guide Run puppet-lint http://puppet-lint.com/ TL;DR puppet-lint 2012 CloudBees, Inc. 10All Rights Reserved 11. Puppet manifests are code too Syntax Check TL;DR RSpec-puppet Static Analysis Automated Tests Use RSpec-puppet http://rspec-puppet.comThere are a lot of people confused by the purpose ofthese tests as they cant test the result of themanifest on a live system. That is not the point ofrspec-puppet.Rspec-puppet tests are there to test the behaviour ofPuppet when it compiles your manifests into a catalogueof Puppet resources.Source: http://rspec-puppet.com/tutorial/ 2012 CloudBees, Inc.11All Rights Reserved 12. Puppet manifests are code too Syntax Check Static Analysis Automated Tests Setup Jenkins to run these on every change to thePuppet manifests You are storing your scripts in Version Control? Now you know your Puppet scripts are doing what theyshould do TL;DRJenkins can save myass!2012 CloudBees, Inc. 12 All Rights Reserved 13. Continuous Integration for Puppet i.e. regularly run the puppet scripts against a testenvironment and verify that the result is a validdeployment Should be just part of the End-to-end ContinuousIntegration testing. Use parameterized builds / build promotion so that QAcan re-use to deploy to their test environment.TL;DR I still need tests onmetal 2012 CloudBees, Inc.13All Rights Reserved 14. What tests should we run against metal?Nagios health checks Real tests Pro Pro This is what we will be Verifies that the installedusing in productionapplication works Checks that the High Confidenceapplication is installed Conand alive Hard to automate Should be fast Long duration Con Does not verify that theTL;DRinstalled application Nagios vs Real: useworksboth 2012 CloudBees, Inc. 14All Rights Reserved 15. The test mix lifecycle80 Devs manually check they implemented theQA gets70 feature blessed builds6050ManualDevs arelazy andAutomated40dont run the Functionalfull30 suiteUnit20Nagios10 0Ops havefinal DevCITest StageProdsanitycheck2012 CloudBees, Inc.15 All Rights Reserved 16. The cost of running tests Lost Users14 Staging envcost $$$costs $$$1210 Keep DevsManual = 8in The $$$ Flow 6 4 2Users payus to test 0 RobotDev CI Test StageProd I can keep costs down and productivity up if I have lots of tests for Jenkins to run for me 2012 CloudBees, Inc.All Rights Reserved 16 17. Continuous Deployment is just one step more Let Jenkins manage the push to production serverstoo.TL;DRAutomation = Jenkins Risks? Accidental deployment Solve with build promotion plugin. Breaking production Revert the change and redeploy. Jenkins is actually a solution to this problem, not a cause Security Jenkins has strong security model. Use a second Jenkins instance (if truly paranoid)2012 CloudBees, Inc.17 All Rights Reserved 18. Setting up a Jenkins ServerTips that will make your life easier down the road2012 CloudBees, Inc. All Rights Reserved 19. Invest in good URL If your users cant see Jenkins, much of the benefit islost Make the URL easier to remember http://sca14-3530.sca.cloudbees.com:8080/jenkins/ http://jenkins.cloudbees.com/ 19 20. Share port 80 with other apps Apache reverse proxy Let you run Jenkins in non-root, toobrowserApacheJenkins For Windows IIS7+URL Rewrite+Application Request Routing 20 21. No /jenkins Use virtual host to distinguish multiple apps, notcontext path Jenkinsbrowser Apache / IIShttp://jenkins.cloudbees.com/Redminehttp://redmine.cloudbees.com/ 21 22. Jenkins Home Directory Prepare for disk usage growth Especially when you start to host jobs from people who arent close to you Make sure you can throw more disks at the problem later No need to waste money on 15000rpm SCSI disks But bigger disk is nice 22 23. Plugins for PuppetTheres 530+ plugins for Jenkins which ones do I actuallyneed? 2012 CloudBees, Inc.All Rights Reserved 24. Jenkins Plugins of relevance to Puppet Here is a list of some of the plugins you may/will want: RVM Plugin Runs your entire build (from SCM check out to post-build actions)within the context of an RVM managed environment of yourchoice 2012 CloudBees, Inc.24All Rights Reserved 25. Jenkins Plugins of relevance to Puppet Here is a list of some of the plugins you may/will want: RVM Plugin Warnings Plugin For integration with puppet-lint2012 CloudBees, Inc.25 All Rights Reserved 26. Jenkins Plugins of relevance to Puppet Here is a list of some of the plugins you may/will want: RVM Plugin Warnings Plugin Promoted Builds Plugin Allows you to set up promotion pipelines that are necessary forgetting the control you need with continuous deployment 2012 CloudBees, Inc.26All Rights Reserved 27. Jenkins Plugins of relevance to Puppet Here is a list of some of the plugins you may/will want: RVM Plugin Warnings Plugin Promoted Builds Plugin CloudSmiths StackHammer Plugin Makes testing and deploying stacks of puppet modules mucheasier and more tractable Removes the left-over failed deployment mess hell2012 CloudBees, Inc. 27 All Rights Reserved 28. Jenkins Plugins of relevance to Puppet Here is a list of some of the plugins you may/will want: RVM Plugin Warnings Plugin Promoted Builds Plugin CloudSmiths StackHammer Plugin HTML Publisher Plugin Handy for publishing your puppet RDocTL;DRLots of plugins for puppet2012 CloudBees, Inc.28 All Rights Reserved 29. Show me howOk, so quit showing me fancy slides, how the feck do I do all this2012 CloudBees, Inc. All Rights Reserved 30. Setup Jenkinspuppet module install rtyler-jenkinspuppet apply -v -e "include jenkins" Quit complaining this is a Puppet conference! Oh yeah, rtyler runs on Ubuntu 10.04does not always work quite sowell on other OSmay need tweaks to work onUbuntu 12.04 2012 CloudBees, Inc.30All Rights Reserved 31. 2012 CloudBees, Inc. 31 All Rights Reserved 32. Setup Jenkins (if you dont want to use Puppet) Download http://mirrors.jenkins-ci.org/war/latest/jenkins.war Either Deploy to your favorite Java Servlet Container (Tomcat/Jetty/etc) Use built in java -jar jenkins.war Goto http://localhost:8080/ or wherever your servletcontainer deployed it 2012 CloudBees, Inc. 32All Rights Reserved 33. Or Dont Setup Jenkins (the shameless plug) Just use CloudBees DEV@cloud(Jenkins as a Service) 2012 CloudBees, Inc.33All Rights Reserved 34. Welcome to Jenkins 2012 CloudBees, Inc. 34All Rights Reserved 35. Testing Puppet with JenkinsIf you are reading these tags do a Mexican Wave on three 12 3!2012 CloudBees, Inc. All Rights Reserved 36. 2012 CloudBees, Inc. 36 All Rights Reserved 37. Issues with testing Puppet on metal Puppet needs to run as root. There can be only one version Of Puppet on a machine Of RVM that Puppet will use on a machine Running tests from a clean baseline Side-effects from previous test runs can affect test results Solve with Virtualization Martyrs can use LXC & Snapshotted VolGroups 2012 CloudBees, Inc.37All Rights Reserved 38. Setup Ruby Version Manager (RVM) Requires RVM Jenkins plugin Enable Run the build in a RVM-managedenvironment Best practice is to give each puppet module their ownGemset A good idea is to use the build jobs name2012 CloudBees, Inc. 38 All Rights Reserved 39. Ensure RVM environment has required GemsOne solution Stick a Gemfile file in theroot of the SCM checkout source "http://rubygems.org" gem "puppet", "=2.7.17 gem "ci_reporter", "=1.7.0 gem "rspec-puppet", "=0.1.3 gem "puppet-lint", "=0.1.13" Add a shell buildstep Run bundle installThese fixed versionswork for me with RubyMore than one way to skin 1.8.7this cat, use whatever worksfor you2012 CloudBees, Inc. 39 All Rights Reserved 40. Syntax check Add a shell build step with the following: for file in $(find . -iname *.pp) do puppet parser validate--render-as s--modulepath=modules"$file" || exit 1; done Or do it your ownway! (Think of thecats though)2012 CloudBees, Inc.40 All Rights Reserved 41. Static Analysis Add a shell build step with the following: find .-iname *.pp-execpuppet-lint--log-format"%{path}:%{linenumber}:%{check}:%{KIND}:%{message}"{}; Add a Scan for compilerwarnings Post-Build Action Add Puppet-lint parser to Scan console log 2012 CloudBees, Inc. 41All Rights Reserved 42. RSpec-puppet Configure Rakefile for ci_reporter require ci/reporter/rake/rspec Add a shell Build Step with the following: export CI_REPORTS=results rake ci:setup:rspec spec Add a Publish JUnit test result report Post-BuildAction **/results/SPEC-*.xml 2012 CloudBees, Inc.42All Rights Reserved 43. Documentation Add a shell build step with the following: ## Cleanup old docs. [ -d doc/ ] && rm -rf doc/ ## Dummy manifests folder. ! [ -d manifests/ ] && mkdir manifests/ ## Generate docs puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc ## Fix docs to remove the complete workspace from all file paths. if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules" fi; grep -l -R ${WORKSPACE} * | while read fname; do perl -pi -e "s@${WORKSPACE}/@/@g" $fname; done; Add a Publish HTML Reports Post-Build Action doc, index.html, Puppet Docs2012 CloudBees, Inc. 43 All Rights Reserved 44. Deploying to metal Set up a build promotion process2012 CloudBees, Inc. 44 All Rights Reserved 45. Promotion process ideas Use Promote immediately once thebuild is complete to deploy into thedevelopers sandbox environment Use Only when manually approvedto allow QA to select which builds totest Or Dev sign-off on being feature complete Use Only when manually approved(Manual tests) with When thefollowing downstream projects buildsuccessfully (Automated tests) topromote to ops2012 CloudBees, Inc. 45 All Rights Reserved 46. Case study: Lookout Jenkins is internal, off the production network 100% Use Capistrano for deploying to production Shipping code git push production-puppet-master:/srv/git/puppet.git ssh production-puppet-master "(cd /etc/puppet/blah && git fetch origin && git checkout ${TAGNAME})" ssh production-puppet-master "run-puppet.sh" 2012 CloudBees, Inc. 46All Rights Reserved 47. CloudSmith can help with on metal testing 2012 CloudBees, Inc.47All Rights Reserved 48. Useful links All the code snippets from this deck http://goo.gl/kNgyu Others links http://cloudbees.com/ http://jenkins-ci.org/ http://rvm.io/ http://puppet-lint.com/ http://rspec-puppet.com/ http://gembundler.com/ https://wiki.jenkins-ci.org/display/JENKINS/RVM+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Warnings+Plugin https://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin 2012 CloudBees, Inc.48All Rights Reserved 49. 2012 CloudBees, Inc. All Rights Reserved