effective devops - velocity new york 2015

Effec%ve'Devops:Collabora'on)and)Tools

Jennifer'Davis'&'Katherine'Daniels

✨ @beerops(@sigje(#effec/vedevops✨ 1

Jennifer'Davis• So$ware)Engineer,)Chef)So$ware

• Co2author)of)"Effec8ve)Devops"

• Founder)of)CoffeeOps

• DevOpsDays)Silicon)Valley)Organizer


Katherine)Daniels• Senior(Opera,ons(Engineer,(Etsy

• Co4author(of("Effec,ve(Devops"

• DevopsDays(NYC(Organizer

• Ship(Show(Podcast(Co4host


Communica)on• Jennifer(DavisTwi/er:(@sigjeEmail:([email protected]

• Katherine(DanielsTwi/er:(@beeropsEmail:([email protected]

Twi$er:(#effec,vedevops


Feedback• Construc*ve-feedback

• What-did-you-find-helpful?

• What-would-you-like-to-see-more/less-of?

• Was-there-anything-you-found-unclear?


Schedule• Introduc*on+to+teams,+devops+principles

• 10:30911:00am+Morning+Break+9+Su?on+Complex

• Visualiza*on+of+work,+Git,+Infrastructure+automa*on

• 12:3091:30pm+Lunch+9+Americas+Halls+1+&+2


Schedule• Tes%ng(infrastucture(automa%on(and(other(changes

• 3:0073:30pm(A:ernoon(Break(7(Su>on(Complex

• Measuring,(monitoring,(and(wrap7up

• Ignites:(Beekman(Parlor((with(Katherine)


Network(Connec,vity

Network(Name:(VelocityAccess(Code:((none)

Local&Downloads:&h.p://10.10.32.101


Expecta(ons• Safe&space&to&share&experiences,&learn&from&each&other

• Code&of&Conduct

• Learn&effec9ve&workflows&for&using&and&tes9ng&source&control&and&configura9on&management


http://www.oreilly.com/conferences/code-of-conduct.html

Team%Introduc.ons

• Meet%your%team!

• What%are%mo/va/ons?%What%are%current%beliefs?%What%are%current%skills?%Gaps%in%skills?

Time:&20&minutes


What%is%Devops


What%is%Devops

Cultural'movement'that'seeks'to:

• change(how(individuals(work,

• value(the(diversity(of(work(done,

• develop(conscious(decisions(in(accelera8on,

• plan(for(scale,(and

• measure(the(effect(of(social(and(technical(change.


Folk%Models• general(popularly(understood(meaning(par2cular(to(a(socio4cultural(grouping(but(which(has(not(been(formally(defined(or(standardized.


Why$Devops?


High%Performing%Devops%Teams

are$more$agile

30X$more$frequent$deployments

8000X$faster$lead$-mes$than$peers

2014%PuppetLabs%State%of%DevOps%Survey


High%Performing%Devops%Teams

are$more$reliable

2X#change#success#rate

12X$faster$mean$-me$to$recovery$(MTTR)

2014%PuppetLabs%State%of%DevOps%Survey


Five%Pillars%of%Effec0ve%Devops

• Collabora(on

• Hiring

• Affinity

• Tools

• Scaling


Collabora'on• Individuals+Working+Together


Hiring• Choosing)Individuals


Affinity• From&Individuals&to&Teams


Tools• Accelerators+of+Culture


Scaling• Evolu'on,*growth*and*advancement*of*the*organiza'on


Collabora'on)and)Tools


Recognizing)your)Devops)Narra3ve


The$Devops$Compact• shared(mutual(understanding

• established(boundaries


Team• Common&purpose

• Defined&beliefs

• Empowered


Careless'Conversa+ons'(inspired'by'Alan'Cyment)

• Pair&up

• Select&one&person&to&go&first.

• For&one&minute,&speaker&talks&about&something&passionate&about.&Listener&stays&seated,&quiet,&and&acts&disinterested.

• Switch&roles&and&repeat.

• Repeat&un@l&both&people&have&done&this&twice.

Time:&5&minutes✨ @beerops(@sigje(#effec/vedevops✨ 27

Careless'Conversa+ons

• How%does%it%feel%not%to%be%listened%to?

• How%does%it%feel%to%ignore%someone?


Cul$va$ng)Empathy

• Collect(stories

• Listen

• Circle(back


Smarter'Teams'Build'Be/er'Value1

• Lots&of&Communica/on

• Contribute&equally&to&team's&discussions

• Theory&of&Mind

• Increased&diversity

1"Engel,"David"et"al."'Reading"The"Mind"In"The"Eyes"Or"Reading"Between"The"Lines?"Theory"Of"Mind"Predicts"CollecCve"Intelligence"Equally"Well"Online"And"FaceIToIFace'."PLoS"ONE"9.12"(2014):"e115212."Web."26"May"2015.


A"life"becomes"meaningful"when"one"sees"himself"or"herself"as"an"actor"within"the"context"of"a"story.

!!"George"Howard


Small%vs%Large%teams• Large'teams'+'roles'may'be'highly'segregated

• Small'teams'+'one'person'may'be'responsible'for'many'roles


Cri$cal(Habits(for(Teams• Code&Review

• Pairing


Code%Review• Max%90%minutes%in%one%se0ng


Pairing• Agile'so*ware'development

• 2'people'work'together'on'1'worksta8on

• Driver':'writes'code

• Observer':'reviews'each'line

• Roles'switch'frequently


Types&of&Pairing• Expert(expert

• Expert(novice

• Novice(novice


Visualiza(on+of+Work• Bug/issue)queue

• Kanban


Factors(for(Success• Management)buy)in

• Training

• Minimize)'pushing'

• Team)sync


Workflow

Work%that%is

• orchestrated

• repeatable

• organized

• moves4from4one4state4to4another


WIP• Work&in&Progress

• work&that&has&had&money&or&people&applied&to&it.


Work%Iden*fica*on• name

• start)date

• end)date

• current)state

• descrip/on

• priority


Task%Handling• What&is&it?

• Can&you&do&anything&with&it?

• What&is&the&next&step?


Projects• Same&requirements&as&a&task

• Larger&in&scope

• May&be&comprised&of&more&than&one&task


Interrupts• Non%planned%work%that%comes%in

• customer%request

• incident

• request%for%help%from%coworker

• single%point%of%knowledge%(you)%work

• high%priority%task%push%from%manager


Blocked(work• Work&that&can&progress&no&further:

• dependent&teams&5&blocked&by&external&team

• insufficiently&qualified&request&5&blocked&by&requester

• dependent&on&SPOK&5&blocked&by&team

• Bme&dependent


Team%Exercise

Discuss&with&your&team:

• What&is&the&difference&between&a&task&and&a&project?

• Do&you&have&interrupts?&What&are&they?

• How&do&you&determine&when&work&is&done?

Time:&15&minutes


Kanban• Start&with&what&you&do&now

• Agree&to&incremental,&evolu7onary&change

• Respect

• Everyone&is&a&leader


Kanban%Prac)ces• Visualize

• Limit-WIP

• Manage-flow

• Make-policies-explicit

• Implement-feedback-loops


Visualize• Intent

• Alignment

• Coherence


Limit%WIP• Pull%(don't%push)


Manage&Flow• Monitor/measure/report

• Incremental2change


Make%Policies%Explicit• Document*processes

• Group*signoff


Implement(Feedback(Loops• Collabora(on

• Retrospec(ves


Devops'Tools• Establish*local*development*environment

• Version*control

• Manual*7>*Automa:on*7>*Con:nuous

• Ar:facts

• Infrastructure

• Sandbox


Local&Development&Environment&(LDE)• Consistent)set)of)tools)across)the)team

• Ability)to)quickly)onboard)new)engineers


Provisioned*Node*,*LDE• AWS%instance%node

• Chef%DK

• Test%Kitchen

• Ruby

• ChefSpec,%ServerSpec

• Git


Configura)on*Management• Process(of(iden-fying,(managing,(monitoring,(and(audi-ng(a(product(through(its(en-re(life(including(the(processes,(documenta-on,(people,(tools,(so8ware,(and(systems.


Version(Control• Records)changes)to)files)or)sets)of)files)stored)within)the)system

• Enable)revisions

• Integrity)checking

• Collabora<on


Ar#fact(Repository• Secure

• Trusted

• Stable

• Accessible

• Versioned


Introduc)on*to*Lab*1Lab$1

User:&chefPassword:&chef


https://github.com/iennae/ed-training/blob/master/labs/assign_1.md

Lab$1

Lab$1

Time:&15&minutes



Lab$2

Lab$2

Time:&15&minutes



Welcome'Back• git%review%


git$pull

git$pull$REMOTE


git$push$remote$branch• transfer)commits)from)a)local)repo)to)a)remote)repo.

• counterpart)to)git)fetch


Infrastructure• Aggregate(of(applica/ons,(configura/ons,(access(control,(data,(compute(nodes,(network,(storage,(processes,(and(people.


Infrastructure+Automa/on• Systems(that(reduce(the(burden(on(people(to(manage(services(and(increase(the(quality,(accuracy(and(precision(of(a(service(to(the(consumers(of(a(service


Infrastructure+Automa/on+Tools• Chef

• Puppet

• Ansible

• Salt

• CFEngine


Introduc)on*to*Chef


Resources• Ingredients+of+infrastructure

• Basic+building+blocks


Resource(Declara,on

RESOURCETYPE "RESOURCE_NAME" do PARAMETER PARAMETER_VALUEend


Example(Resource(Type(1(package

A"package"to"be"installed:

package "httpd" do action :installend


Example(Resource(Type(1(service

A"service"that"should"be"started:

service "httpd" do supports :restart => :true action [:enable, :start]end


Resources

A"resource"is"a"statement"of"policy"that:

• Describes)the)desired)state)for)an)element

• Specifies)a)resource)type888such)as)package,)template,)or)service

• Lists)addi;onal)details)(also)known)as)parameters),)as)necessary

• Are)grouped)into)recipes


Recipes• Collec'on)of)ordered)resources

• Combina'on)of)ruby)and)Chef)DSL


Cookbooks• Thema'c

• Collec'on-of-recipes-and-other-suppor'ng-files


Roles• Abstrac)on,describing,func)on,of,system

• Name

• Descrip)on

• Run,list,(ordered,list,of,recipes,and,roles)


Run$List• Ordered&list&of&recipes&and&roles

• Specific&to&a&node


Nodes• Machine)(virtual,)physical,)cloud)server,)or)other)device))that)is)managed)by)Chef


Environments• Abstrac)on,models,workflow

• Name

• Descrip)on

• Cookbook,version,pinning


Supermarket• Community*site*with*a*number*of*cookbooks

• Read*before*using*in*your*environment


Chef%DK• Chef&development&kit

• Includes&a&number&of&u8li8es&and&so9ware&to&facilitate&cookbook&crea8on

• Free&download&off&of&the&website


Berkshelf• Dependency)management

• Included)with)Chef)DK


Test%Kitchen• Included)with)Chef)DK

• Sandbox)automa8on

• Test)harness


Test%Kitchen• Execute(code(on(one(or(more(pla1orms

• Driver(plugins(suppor7ng(various(cloud(and(virtualiza7on(providers


.kitchen.yml• driver

• provisioner

• pla-orms

• suites


.kitchen.yml,driver• virtualiza*on-or-cloud-provider

Example:)vagrant,)docker


.kitchen.yml,provisioner• applica'on*to*configure*the*node

Example:)chef_zero


.kitchen.yml,pla/orms• target'opera*ng'systems

Example:)centos/6.5


.kitchen.yml,suites• target'configura-ons

Example:

name:default run_list: - recipe[apache::default] attributes:


Kitchen(commands((1/2)• kitchen)init

• kitchen)list

• kitchen)create

• kitchen)converge


Kitchen(commands((2/2)• kitchen)verify

• kitchen)destroy

• kitchen)test


Docker• Images

• Registries

• Containers


Lab$3

Time:&30&minutes


Managing&Risk• Test

• Small*frequent*releases


Lin$ng• Ensure(code(adheres(to(styles(and(conven2ons

• Weave(expecta2ons(into(development

• Encourages(collabora2on


Tes$ng• Documen)ng+objec)ves+and+intent

• Measuring+"done"


Code%Correctness• foodcri(c

• rubocop


Integra(on*Tests• ServerSpec


Rubocop• Ruby&linter

• Ruby&style&guide

• Included&with&ChefDK


https://github.com/bbatsov/ruby-style-guide

Rubocop'Example$ rubocop cookbooks/COOKBOOK1 cookbooks/COOKBOOK2 cookbooks/COOKBOOK4


Reading(Rubocop(OutputInspecting 8 filesCWCWCCCC

• ."means"that"the"file"contains"no"issues

• C"means"a"issue"with"conven2on

• W"means"a"warning

• E"means"an"error

• F"means"an"fatal"error✨ @beerops(@sigje(#effec/vedevops✨ 105

Disabling)Rubocop)cops

Any$configura,on$in$.rubocop.yml$is$disabled.

To#disable#string#literals:

StringLiterals: Enabled: false


Foodcri'c• Chef&linter

• Chef&style&guide

• Included&with&ChefDK


http://foodcritic.io

Foodcri'c(Example

$ foodcritic cookbooks/setup


Reading(Foodcri-c(OutputFC008: Generated cookbook metadata needs updating: ./metadata.rb:2


ServerSpec• Tests&to&verify&servers&func0onality

• Resource&types

• Package,&service,&user,&and&many&others

• Integrates&with&Test&Kitchen

• h?p://serverspec.org


ServerSpec'Generic'Form

describe "<subject>" do it "<description>" do expect(thing).to eq result endend


ServerSpec'Poten,al'Tests• Is$the$service$running?

• Is$the$port$accessible?

• Is$the$expected$content$being$served?


ServerSpec'Example

describe 'apache' do it "is installed" do expect(package 'httpd').to be_installed end it "is running" do expect(service 'httpd').to be_running endend


Reading(ServerSpec(Outputapp::default httpd service is running

Finished in 0.26429 seconds (files took 0.7166 seconds to load)1 example, 0 failures


Lab$4

Time:&30&minutes


Lab$5

Time:&20&minutes


Test,&Monitor,&or&Diagnos/c2

1. Where'is'it'going'to'run?

2. When'is'it'going'to'run?

3. How'o4en'will'it'run?

4. Who'is'going'to'consume'the'result?

5. What'is'the'en;ty'going'to'do'with'it?

2"Lam,"Yvonne."'Sysadvent:"Day"5"6"How"To"Talk"About"Monitors,"Tests,"And"DiagnosCcs'."Sysadvent.blogspot.com."N.p.,"2014."Web."26"May"2015.


Measuring*Impact*and*Value*of*Change


Impact'of'Change


Impact'on'Availability

• Overall(site/app(availability

• Individual(service(availability


Availability)Monitoring

• Up$me:

• Pingdom,/Moni$s,/Uptrends,/etc

• Ver$cal/Line/Technology:

• Availability/a?er/deploys/changes


Even%nator


Service'Availability

• Nagios:)Service/level)monitoring)and)aler5ng

• Nagios/herald:)Alert)context

• OpsWeekly:)Historical)alert)data


Nagios


define command { command_name check_mongodb_query command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -q $ARG5$}

define service { use generic-service hostgroup_name Mongo Servers service_description Mongo Connect Check check_command check_mongodb!connect!27017!2!4}


define servicedependency{ host_name WWW1 service_description Apache Web Server dependent_host_name WWW1 dependent_service_description Main Web Site execution_failure_criteria n notification_failure_criteria w,u,c}


Nagios'herald


OpsWeekly


Impact'on'Quality

• Service(quality((SLAs)

• Visibility(of(quality


Statsd


>>> import statsd>>>>>> timer = statsd.Timer('MyApplication')>>>>>> timer.start()>>> # do something here>>> timer.stop('SomeTimer')


>>> import statsd>>>>>> counter = statsd.Counter('MyApplication')>>> # do something here>>> counter += 1


>>> import statsd>>>>>> average = statsd.Average('MyApplication', connection)>>> # do something here>>> average.send('SomeName', 'somekey:%d'.format(value))


Graphite


Value&of&Change


Value&of&Availability

• Be$er&for&customers

• Be$er&for&employees&(internal&services)

• Fewer&pages


Value&of&Quality

• Deploys)take)less)-me

• Also)be1er)for)customers

• More)visibility)into)issues


Retrospec)ve+!


Review• Recognizing*your*Devops*Narra4ve

• Applica4on*Deployment*Planning

• Infrastructure*as*code

• Introducing*repeatable,*testable*change

• Measuring*impact*and*value*of*change


Next%Steps• Manual,(Automa-on(to(Con-nuous("X"

• Be(the(storylistener(and(storyteller(in(your(org

• Effec-ve(Devops(available(in(Early(Release


Thank&you!&❤@sigje

@beerops


https://twitter.com/sigje

https://twitter.com/beerops