the unintended benefits of chef
DESCRIPTION
When most people talk about automating infrastructure, they focus on things like consistency, scalability, and flexibility. While fine goals, we recently converted several projects to Chef for both systems AND application deployment, and found that, with a little work, these tools could also help you enable better software quality assurance, load modeling, and even improve resource allocation. By sharing cookbooks across projects, we were able to standardize practices and eliminate arbitrary differences, while using parameterization to perfectly isolate the special needs of each project. This allowed us to transfer knowledge among staff much more quickly. Pulling in and parameterizing application state – database contents, website assets, uploaded content – allowed us to spin up new environments with as much or as little state as needed. Integrating with Vagrant and Jenkins, we were then able to use chef to treat the entire image – system and application – as a test fixture. As each engineer (ops or dev) has visibility into the whole stack, we can more easily move people between dev and ops, or between projects.TRANSCRIPT
The Unintended Benefits of Chef
Standardization FTW
whoamiClinton Wolfe, Web Architect @ OmniTI mod perl developer since 1996cheffin' since 2012
@clintoncwolfegithub: clintoncwolfe
LEGO group
$JOBomniti.comcustom web development - UX to hostingwe make OmniOS, an Illumos based OS distro
our dev clientsmostly brownfield worknew featuresoh no it won't scalemy nephew built it
client scalesome have hundreds of nodesmost clients have < 10 nodes
some have many, want fewersome have few, need morewe specialize in doing more with less
mijasper/brickshelf
process : anti-processeach customer has "unique" problems
some are, most aren't
we're flexible, urging gradual change
TMTOWTDI....dev environmentsasset statecronself-referential urlsCPAN/PECL/gem/egg/npm builds
“You Only Deploy Once!"
clients: YODO!!!
except for disaster recovery
clients: YODO!!!
except for disaster recoveryexcept for onboarding devs
clients: YODO!!!
except for disaster recoveryexcept for onboarding devsexcept for CI
clients: YODO!!!
except for disaster recoveryexcept for onboarding devsexcept for CIexcept for scaling out
clients: YODO!!!
except for disaster recoveryexcept for onboarding devsexcept for CIexcept for scaling outexcept for load testing
clients: YODO!!!
except for disaster recoveryexcept for onboarding devsexcept for CIexcept for scaling outexcept for load testingexcept for upgrade testing
clients: YODO!!!
except for disaster recoveryexcept for onboarding devsexcept for CIexcept for scaling outexcept for load testingexcept for upgrade testingexcept for datacenter moves
QA is hardcan't make new QA environments
QA env is precious
state hard to manage
If testing is hard, no testing is done
mijasper/brickshelf
surely we can do better
We want standardization
Client wants deployability (scale-out, resilience)
Chef FTW
The Plan!Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early
cheffing as discoveryExploring the existing app for on-boarding
Even rsyncing a golden image is OK
capture it in chefGet deployable (in Vagrant ) ASAP mijasper/customminifigs.co.uk
The Plan!Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles Automate testing early
app deployment in chef?
most app deployment tools partially dictate layout, push process, etc
who knows what we have been handed
but chef can do pretty much anything!
cheffing an app
identify and decouple rolessetup user auth, systems stuffcode checkouttemplatize config filesservice definitions
Iterate, standardize, simplify!
The Plan!Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early
finding variances
understand existing variancesstandardize the arbitraryjustify the ones that need to remain
writing a custom cookbook is a red flag
jokeith/brickshelf
Our shared cookbooks
Over 20 cookbooks used internally at the company, cross-clientAttribute-driven default recipesExposes project variances in clear relief... yet we can easily extend them if needed
Shared roles/handlers/databags, too
it's still OK to punt
We will never be fully standardizedAs old projects mature, new projects arrive
known technical debtcan be healthy
mijasper/brickshelf
The Plan!Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early
cheffing: ci with jenkins
static checks on chef configscratchbuild Vagrant runsparametric builds and Vagrantfilesvagrant-test-subject
mijasper/brickshelf
system tests
High-level, near-english behavioral tests
Not low-level unit tests against codebase Supposed to be serving http? Check for it!
Easy to add more later - start with minimum
aside: our test harness
vagrant-rspec-ci gemprovision ok?services running?connect to port?Capybara for web UI testingJunit output for pretty Jenkins graphs
Test spec exampledescribe "TrafficServer Service" do
before(:all) do
@vm = VagrantTestSubject::VM.attach()
end
it "should appear as a healthy service" do
@vm.should have_running_service("trafficserver")
end
it "should be listening on external_ip:80" do
@vm.should be_listening_on_external_ip(80)
end
it "should be the right process name on port 80" do
process = @vm.process_name_listening('127.0.0.1', 80)
process.should match(/\/opt\/ts\/bin\/traffic_manager/)
end
it "should respond with HTTP 200 to / on port 80" do
@vm.http_get('/').should be_http_not_found # from rspec-http gem
end
end
Shiny, Happy TeamsOps Team ImpactDev Team ImpactQA Team ImpactOrganizational Impact
LEGO Group
Ops Team ImpactChef everywhere! YAY!
Chef for everything, especially wildly divergent app deployments! BOO!
Direct exposure to dev team deployment needs & tooling... a more devvy ops team
Dev Team ImpactCan get a new dev env anytime, have more than one - YAYSame build everywhere – YAYI have to learn ruby? BOOSome awareness of practices on other projects, improved perspectiveMuch more visibility into how systems are provisioned – a more opsy dev team
QA Team ImpactUse vagrant VMs as test subjects
QA VM provisioning can now be elastic, responsive to testing/release cycles
Dev and QA exposure to systems BDT
Dev and ops teams more QA-y: devopsqa!
the VM as fixture
data loads - vary by test scenariodata loads - vary for scaleasset loadsrole permutationsos, vm, env permutationsclusters of related machines (db, web, CDN, client)
aside: loading fixturesWe use Chef roles to define attributes that load fixture state
We use recipes to read the attributes and converge to the needed test state
•special purge-all role clears all state•use env var to pass fixture list into vagrant for
provision
i accidentally the whole dbfixture loading must be used with care
use a safety flag to enable it
Nelson Yrizarry/brickshelf
Organizational Impact
Standardization eases: actual deployment on-call ops can support the app more devs responsible for deployability ops can participate in app management reduced ramp-up time onto existing projects
