an almost complete continuous delivery pipeline including configuration management
DESCRIPTION
How we have created a build pipeline for continous delivery at Recorded Future. This includes also test of Chef cookbooks and configuration. Presentation at Config Management Camp 2014TRANSCRIPT
![Page 1: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/1.jpg)
An Almost Complete Continuous Delivery Pipeline Including Configuration Management
Ulf Månsson @ulfmanssonRecorded Future AB
![Page 2: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/2.jpg)
Why almost complete?
You never complete
It's not easy
![Page 3: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/3.jpg)
![Page 4: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/4.jpg)
Recorded Future
![Page 5: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/5.jpg)
![Page 6: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/6.jpg)
We wanted
● Quick feedback
● Build all artifacts
● Upload all build artifacts to s3
● Trust in deploying every successful build
● Test everything
![Page 7: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/7.jpg)
What do we mean test everything?
● Unit tests
● Integration tests
● End-to-end test
● UI test
● Performance test
● Cookbook tests
● Deployment tests
● Configuration tests
![Page 8: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/8.jpg)
![Page 9: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/9.jpg)
![Page 10: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/10.jpg)
![Page 11: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/11.jpg)
No champagne
![Page 12: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/12.jpg)
![Page 13: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/13.jpg)
![Page 14: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/14.jpg)
Long journey
● The “build master” left the company
● Lacking focus
● Very long build times
● Use the build pipe line plugin
![Page 15: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/15.jpg)
Continues delivery embrace devops
● Everyone should dare to touch stuff
● Developers touch ops code
● Operation touch application code
● Everyone should be able to deploy
![Page 16: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/16.jpg)
operations@recordedfuture
● Quite big team
● Infra code
● Build pipe line
● Service and on premise installations
● End to end tests
● Performance tests
● Amazon operation costs
![Page 17: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/17.jpg)
Our system
● A number of major infrastructure components – ElasticSearch, RabbitMQ etc
● About 45 different processes
● Development in Java, Python, Scala and Ruby
● External API and web servers
● Developed by 5 different teams in Sweden, US and Ukraine
![Page 18: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/18.jpg)
Subversion
For good and bad
All code is code
Keep the code in the same repository
Configurations also in subversion
Treat all code in the same way, it doesn’t matter if it’s application code, infra code, test code or whatever
![Page 19: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/19.jpg)
![Page 20: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/20.jpg)
![Page 21: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/21.jpg)
Feature toggling to our help
● No branching
● One trunk
● Took some time train developers
● Now we try to train infra coders as well – Feature toggling in Chef
● Read more by Jez Humble http://continuousdelivery.com/2011/07/on-dvcs-continuous-integration-and-feature-branches/
![Page 22: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/22.jpg)
Jenkins
and his slaves...
![Page 23: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/23.jpg)
Build Flow Plugin the thing!
out.println "Start"
revision=build.properties["environment"]["SVN_REVISION"]
out.print "Original revision for this build is "
out.println revision
parallel (
{build("master-build-flow",SVN_REVISION: revision)},
{ ignore(FAILURE) {build("master-infrastructure_test",SVN_REVISION: revision)}},
{build("master-fast_track_upload_cookbooks-flow",SVN_REVISION: revision)},
{ ignore(FAILURE){build("master-infrastructure_lint",SVN_REVISION: revision)}}
)
build( "master-fast_track_deploy-flow", SVN_REVISION: revision)
build( "master-fast_track_integration_test_harvesting-flow", SVN_REVISION: revision)
parallel (
{ build("master-fast_track_test_ff-flow", SVN_REVISION: revision) },
{ build("master-fast_track_test_ie-flow", SVN_REVISION: revision) }
)
![Page 24: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/24.jpg)
Build Graph View Plugin
![Page 25: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/25.jpg)
Jenkins text finder
puts "The build is UNSTABLE" if failed
![Page 26: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/26.jpg)
S3
● Store artifacts
● Simple
● Control
● Reliable
● Accessible
![Page 27: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/27.jpg)
![Page 28: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/28.jpg)
![Page 29: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/29.jpg)
Test tools
● Test kitchen
● Server spec
● Maven
● Cucumber
● Rspec
● Load impact
● Selenium
![Page 30: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/30.jpg)
Build tools
Then: Only Maven
Now: Mix of Rake and Maven
![Page 31: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/31.jpg)
Environment cookbook
# Cookbook Name:: env
# Recipe:: fast_track_1 (test environment)
include_recipe 'env::prod'
node.default[:rabbitmq][:version] = "3.1.5"
# Interactive
node.default['rfwebapp']['interactive']['java_opts'] = 'Xms1G Xmx1G'
In env::default include_recipe "env::#{node.chef_environment}"
![Page 32: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/32.jpg)
Why environment cookbook
● We can version handle it
● We are sure that the test environment will get the same settings as production environment
● On premise installation use chef-solo
● When we upload the cookbooks the environment settings will be uploaded as well
![Page 33: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/33.jpg)
Our master build pipe line
![Page 34: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/34.jpg)
Masterflow
● Only the build-flow plugin, describes the build
● Triggered by subversion polls
● Co-ordinate the sub build processes
● Only one at a time running
![Page 35: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/35.jpg)
Master build flow
● Compiles all application code
● Run unit tests
● Builds application artifacts
● Uploads artifacts to S3
![Page 36: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/36.jpg)
Upload cookbooks
● Upload cookbooks to chef server for the build test environment - fast_track
![Page 37: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/37.jpg)
Infrastructure lint
● Food critic of course
● Tailor – nope
● Rubocop – probably as next step
![Page 38: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/38.jpg)
Fast track...
● All-in-one box
● Will deploy,configure and launch all processes
● Configure infrastructure components
● Spin-up EC2 instance
![Page 39: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/39.jpg)
Fast track cont...
● Use Chef server
● Deploy all processes with Chef
● Instance used for following tests
● Perfect case for Docker
![Page 40: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/40.jpg)
Infrastructure test
● Test-kitchen tests
● Mainly server-spec
● Run in lxc with vagrant-lxc
● Run our “default” suite in test-kitchen
● Disabled in the Master-flow at the moment
![Page 41: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/41.jpg)
Endtoend test
● Cucumber tests
● Populate system with data
● Tests via our API
● Check process log database to ensure all steps has been run
![Page 42: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/42.jpg)
Browser tests IE and Firefox
● Run via Selenium
● Test browser functionality
● Upload screen snapshots from the tests to s3
![Page 43: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/43.jpg)
Supporting tools
![Page 44: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/44.jpg)
Bygge – an overview
![Page 45: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/45.jpg)
Logstash
● Collect all log from the fast track run
● Used for problem solving
● Easy to access with a link from Bygge
● We also use it in production
![Page 46: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/46.jpg)
Sensu monitoring
● Almost same monitoring as in production of each fast truck run
● Easy to access monitoring dashboard and see errors
● Will fail builds when we have checks that fails
![Page 47: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/47.jpg)
Lessons learned
![Page 48: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/48.jpg)
Opscode jenkins cookbook
● Really good
● Providers for slaves, jobs, plugins
● Updated
![Page 49: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/49.jpg)
Run parallel
Many and smaller build steps
![Page 50: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/50.jpg)
![Page 51: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/51.jpg)
From 150 minutes to 45 minutes in 3 days
● SSD instances for Jenkins
● SSD test instances
● Build slaves
● Parallel execution
● Pre upload
![Page 52: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/52.jpg)
![Page 53: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/53.jpg)
Infrastructure
Jenkins
Logstash
Sensu
Fast track instancesFast track instancesFast track instances
![Page 54: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/54.jpg)
I would like to share experience
![Page 55: An almost complete continuous delivery pipeline including configuration management](https://reader034.vdocuments.site/reader034/viewer/2022042613/54b762ae4a7959f71f8b46f4/html5/thumbnails/55.jpg)