So Continuous. Much Delivery.

Very Chef. Wow.

A Case Study on using Chef to start building a Continuous Delivery Pipeline

• George Miranda

• Sr Consultant at Chef Software, Inc.

• Unix guy (15+ years)

About Me

Minimum Viable Pipeline

• Step 1: Develop a new change

• Step 2: ???

• Step 3: Production!

• MOAR FASTERZ

What we know

• Must utilize existing tools within the company

• Git for SCM

• Jenkins approved for use

• Working in a static VM environment

• Just migrated to single cookbook repos

• Starting with infrastructure cookbooks

• Want a manual go-to-production button (ugh!)

Case Study: Requirements

• Git PR model: branch from master for any new feature

• 4-person team, only 3 active at any time

• Code review done manually and informally

• Simple communication/reqs (makes it easy!)

Case Study: Code Review Model

• How are developers expected to work locally?

• When do they push to remote? How do we verify their work?

• Code Review criteria: what does it mean to be ready to merge?

• How do we go from merged code to artifact?

• How do we get that artifact all the way to Production?

Figuring out new workflow

• New branch for every feature

• Create a failing test

• Write a resource to pass the test

• Local commits

• Test-Kitchen + guard

• Once local tests passed, push to remote

Local Development Work

• Open a Pull Request (new branch to master)

• Triggers a build via Jenkins GHPRB plugin

Push to remote

The Verify Build Job

• Verify syntax (knife cookbook check)

• Foodcritic Rules

• Test-Kitchen w/ BATS busser

Push to remote

@test "My directory is created" {! test -d /foo/bar!}!!@test "A basharific test" {! if [ foo != bar ]; then! skip "foo isn't bar"! fi!! run foo! [ "$status" -eq 0 ]!}!!• https://github.com/sstephenson/bats • Super low learning curve (but also very limited)

BATS: Simple Unit Tests

• If failed, notify

• Another commit to the same branch triggers another Verify Build Job

• Super easy to track, comment, and approve

• If passed, let’s go to Human Code Review

Push to remote

• Only one change per one cookbook at one time

• Must have test for feature that changed

• One for one: resource unit tests

• Consider the smoke test

Human Code Review Rules

• Unit tests: small, fast, check one single concern

• In this context: checking Chef resources

• Smoke tests: test multiple things in the course of one concern

• In this context: check the intent of a recipe

• Note: that was testing for this use case

Unit Test vs Smoke Test

• Only 3 active team members at any given time

• Submitter cannot approve

• Merge approval requires 2 approvals

• Code review can happen at any time, but only merge when you’re ready to fix it.

When are we ready to merge?

• Freeze your cookbooks!

• Semantic versioning: Major.Minor.Patch

• You own Major.Minor

• The Pipeline owns .Patch

• No one gets to knife upload

No one.!

Ever.!

• "git merge" is the new "knife upload"

Merged code to artifact

• Bumps Cookbook version

• Re-commits to master

• Upload frozen cookbook (via berks)

• Pin that new cookbook to the Integration environment

• Converge all nodes that use that cookbook

The Integration Job

• First sign that things may be broken

• These nodes also run smoke tests

• serverspec, minitest, etc

The Integration Job

• We survived! Trigger the next job(s)

• The Jenkins Build Pipelines Plugin allows upstream/downstream definitions to string together jobs

• From here out, it’s all the same Promote Job*

• After the Integration job, we just run X number of Promote Jobs

The Integration Job

* (mostly)

• Pin cookbook to new Chef Environment

• Converge all nodes using this cookbook

• Run Tests

Promote Jobs

#!/opt/chef/embedded/bin/ruby !require 'chef/environment' require 'chef' Chef::Config.from_file("/var/lib/jenkins/tools/knife.rb") !def pin_env(env, cookbook_versions) to = Chef::Environment.load(env) cookbook_versions.each do |cb, version| puts "Pinning #{cb} #{version} in #{env}" to.cookbook_versions[cb] = version end to.save end !cookbook_data = Array.new !if File.exists?(File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb'))) metadata_file = File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb')) File.read(metadata_file).each_line do |line| if line =~ /^name\s+["'](\w+)["'].*$/ cookbook_data << $1 end if line =~ /^version\s+["'](\d+\.\d+\.\d+)["'].*$/ cookbook_data << "= #{$1}" end end end !cookbook_versions = Hash[*cookbook_data] !pin_env(ARGV[0], cookbook_versions)

Pin the cookbook to Env

$ berks apply <environment>

Pin the cookbook to Env

$ knife ssh "recipes:mycookbook AND chef_environment:promote-environment” 'sudo chef-client'!

… OR …

Pushy!

Converge Nodes

• Most testing frameworks have a Report Handler to automatically run tests

• chef-serverspec-handler

• minitest-handler

• Deploy to your nodes by adding ‘chef_handler’ to their run_list

• Many community cookbooks are already packaged with tests

Run Tests

• In this particular use case:

• Build job: BATS (unit tests)

• Integration & Promote jobs: serverspec (smoke tests)

• UAT: also ran Cucumber tests (acceptance)

Run Tests

• Can string together N number of promotions

• UAT

• Production A

• Production B

• etc

Promoting to more environments

• In production monitoring is the test

• Could not queue up changes reliably anyway

• There is no spoon

Push to Production

• Small incremental deployments led to greater confidence

• TDD was pushed to the forefront of priorities

• Commitment from Dev group to write application deployment cookbooks

• But the biggest lesson learned…

Results

• Continuous Delivery is a practice, not a tool

• Small incremental changes in code

• Small incremental changes in workflow

• Small incremental changes in tooling

• You will constantly improve your code, your workflow, your tools, your team, and your skills.

Let’s Go Devop with a CD tool

RECAP

• Step 1: Develop a new change

• Step 2: ???

• Step 3: Production!

• MOAR FASTERZ

What We Wanted

• (Pre-req) Test Driven Development

• 2A. Establish development workflow before submitting changes *

• 2B. Auto verification of submission before humans look at it

• 2C. Humans Apply Code Review Criteria *

• 2D. Don’t merge unless you mean it *

• 2E. Merge kicks off an Integration Job

• 2F. Followed by a series of Promotion Jobs

• 2G. There is no spoon *

Wait… what was Step 2?

• Step 1: Develop a new change

• Step 2:

!

!

• Step 3: Production!

• Step 4: Level Up. This is great!

• Step 5: MOAR THINGS! Wait. This is hard!

• Go to Step 1

What We Got

(Pre-req) Test Driven Development

2A. Establish development workflow before submitting changes *

2B. Auto verification of submission before humans look at it

2C. Humans Apply Code Review Criteria *

2D. Don’t merge unless you mean it *

2E. Merge kicks off an Integration Job

2F. Followed by a series of Promotion Jobs

2G. There is no spoon *

• Test Kitchen — http://kitchen.ci/

• Guard Plugin for Test Kitchen — https://github.com/test-kitchen/guard-kitchen

• Foodcritic — http://acrmp.github.io/foodcritic/

• Berkshelf — http://berkshelf.com/

Key Chef Ecosystem Tools

• git

• github

• build-pipeline-plugin

• ghprb

• warnings

• mailer

Helpful Jenkins Plugins

I want to hear from you! !

@gmiranda23 gmiranda@getchef.com