effective devops - velocity new york 2015

Effec%ve'Devops:Collabora'on)and)Tools

Jennifer'Davis'&'Katherine'Daniels

✨ @beerops(@sigje(#effec/vedevops✨ 1

Jennifer'Davis• So$ware)Engineer,)Chef)So$ware

• Co2author)of)"Effec8ve)Devops"

• Founder)of)CoffeeOps

• DevOpsDays)Silicon)Valley)Organizer

Katherine)Daniels• Senior(Opera,ons(Engineer,(Etsy

• Co4author(of("Effec,ve(Devops"

• DevopsDays(NYC(Organizer

• Ship(Show(Podcast(Co4host

Communica)on• Jennifer(DavisTwi/er:(@sigjeEmail:(sparklydevops@gmail.com

• Katherine(DanielsTwi/er:(@beeropsEmail:(sparklyyakshaver@gmail.com

Twi$er:(#effec,vedevops

Feedback• Construc*ve-feedback

• What-did-you-find-helpful?

• What-would-you-like-to-see-more/less-of?

• Was-there-anything-you-found-unclear?

Schedule• Introduc*on+to+teams,+devops+principles

• 10:30911:00am+Morning+Break+9+Su?on+Complex

• Visualiza*on+of+work,+Git,+Infrastructure+automa*on

• 12:3091:30pm+Lunch+9+Americas+Halls+1+&+2

Schedule• Tes%ng(infrastucture(automa%on(and(other(changes

• 3:0073:30pm(A:ernoon(Break(7(Su>on(Complex

• Measuring,(monitoring,(and(wrap7up

• Ignites:(Beekman(Parlor((with(Katherine)

Network(Connec,vity

Network(Name:(VelocityAccess(Code:((none)

Local&Downloads:&h.p://10.10.32.101

Expecta(ons• Safe&space&to&share&experiences,&learn&from&each&other

• Code&of&Conduct

• Learn&effec9ve&workflows&for&using&and&tes9ng&source&control&and&configura9on&management

Team%Introduc.ons

• Meet%your%team!

• What%are%mo/va/ons?%What%are%current%beliefs?%What%are%current%skills?%Gaps%in%skills?

Time:&20&minutes

What%is%Devops

Cultural'movement'that'seeks'to:

• change(how(individuals(work,

• value(the(diversity(of(work(done,

• develop(conscious(decisions(in(accelera8on,

• plan(for(scale,(and

• measure(the(effect(of(social(and(technical(change.

Folk%Models• general(popularly(understood(meaning(par2cular(to(a(socio4cultural(grouping(but(which(has(not(been(formally(defined(or(standardized.

Why$Devops?

High%Performing%Devops%Teams

are$more$agile

30X$more$frequent$deployments

8000X$faster$lead$-mes$than$peers

2014%PuppetLabs%State%of%DevOps%Survey

High%Performing%Devops%Teams

are$more$reliable

2X#change#success#rate

12X$faster$mean$-me$to$recovery$(MTTR)

2014%PuppetLabs%State%of%DevOps%Survey

Five%Pillars%of%Effec0ve%Devops

• Collabora(on

• Hiring

• Affinity

• Tools

• Scaling

Collabora'on• Individuals+Working+Together

Hiring• Choosing)Individuals

Affinity• From&Individuals&to&Teams

Tools• Accelerators+of+Culture

Scaling• Evolu'on,*growth*and*advancement*of*the*organiza'on

Collabora'on)and)Tools

Recognizing)your)Devops)Narra3ve

The$Devops$Compact• shared(mutual(understanding

• established(boundaries

Team• Common&purpose

• Defined&beliefs

• Empowered

Careless'Conversa+ons'(inspired'by'Alan'Cyment)

• Pair&up

• Select&one&person&to&go&first.

• For&one&minute,&speaker&talks&about&something&passionate&about.&Listener&stays&seated,&quiet,&and&acts&disinterested.

• Switch&roles&and&repeat.

• Repeat&un@l&both&people&have&done&this&twice.

Time:&5&minutes✨ @beerops(@sigje(#effec/vedevops✨ 27

Careless'Conversa+ons

• How%does%it%feel%not%to%be%listened%to?

• How%does%it%feel%to%ignore%someone?

Cul$va$ng)Empathy

• Collect(stories

• Listen

• Circle(back

Smarter'Teams'Build'Be/er'Value1

• Lots&of&Communica/on

• Contribute&equally&to&team's&discussions

• Theory&of&Mind

• Increased&diversity

1"Engel,"David"et"al."'Reading"The"Mind"In"The"Eyes"Or"Reading"Between"The"Lines?"Theory"Of"Mind"Predicts"CollecCve"Intelligence"Equally"Well"Online"And"FaceIToIFace'."PLoS"ONE"9.12"(2014):"e115212."Web."26"May"2015.

A"life"becomes"meaningful"when"one"sees"himself"or"herself"as"an"actor"within"the"context"of"a"story.

!!"George"Howard

Small%vs%Large%teams• Large'teams'+'roles'may'be'highly'segregated

• Small'teams'+'one'person'may'be'responsible'for'many'roles

Cri$cal(Habits(for(Teams• Code&Review

• Pairing

Code%Review• Max%90%minutes%in%one%se0ng

Pairing• Agile'so*ware'development

• 2'people'work'together'on'1'worksta8on

• Driver':'writes'code

• Observer':'reviews'each'line

• Roles'switch'frequently

Types&of&Pairing• Expert(expert

• Expert(novice

• Novice(novice

Visualiza(on+of+Work• Bug/issue)queue

• Kanban

Factors(for(Success• Management)buy)in

• Training

• Minimize)'pushing'

• Team)sync

Workflow

Work%that%is

• orchestrated

• repeatable

• organized

• moves4from4one4state4to4another

WIP• Work&in&Progress

• work&that&has&had&money&or&people&applied&to&it.

Work%Iden*fica*on• name

• start)date

• end)date

• current)state

• descrip/on

• priority

Task%Handling• What&is&it?

• Can&you&do&anything&with&it?

• What&is&the&next&step?

Projects• Same&requirements&as&a&task

• Larger&in&scope

• May&be&comprised&of&more&than&one&task

Interrupts• Non%planned%work%that%comes%in

• customer%request

• incident

• request%for%help%from%coworker

• single%point%of%knowledge%(you)%work

• high%priority%task%push%from%manager

Blocked(work• Work&that&can&progress&no&further:

• dependent&teams&5&blocked&by&external&team

• insufficiently&qualified&request&5&blocked&by&requester

• dependent&on&SPOK&5&blocked&by&team

• Bme&dependent

Team%Exercise

Discuss&with&your&team:

• What&is&the&difference&between&a&task&and&a&project?

• Do&you&have&interrupts?&What&are&they?

• How&do&you&determine&when&work&is&done?

Time:&15&minutes

Kanban• Start&with&what&you&do&now

• Agree&to&incremental,&evolu7onary&change

• Respect

• Everyone&is&a&leader

Kanban%Prac)ces• Visualize

• Limit-WIP

• Manage-flow

• Make-policies-explicit

• Implement-feedback-loops

Visualize• Intent

• Alignment

• Coherence

Limit%WIP• Pull%(don't%push)

Manage&Flow• Monitor/measure/report

• Incremental2change

Make%Policies%Explicit• Document*processes

• Group*signoff

Implement(Feedback(Loops• Collabora(on

• Retrospec(ves

Devops'Tools• Establish*local*development*environment

• Version*control

• Manual*7>*Automa:on*7>*Con:nuous

• Ar:facts

• Infrastructure

• Sandbox

Local&Development&Environment&(LDE)• Consistent)set)of)tools)across)the)team

• Ability)to)quickly)onboard)new)engineers

Provisioned*Node*,*LDE• AWS%instance%node

• Chef%DK

• Test%Kitchen

• Ruby

• ChefSpec,%ServerSpec

• Git

Configura)on*Management• Process(of(iden-fying,(managing,(monitoring,(and(audi-ng(a(product(through(its(en-re(life(including(the(processes,(documenta-on,(people,(tools,(so8ware,(and(systems.

Version(Control• Records)changes)to)files)or)sets)of)files)stored)within)the)system

• Enable)revisions

• Integrity)checking

• Collabora<on

Ar#fact(Repository• Secure

• Trusted

• Stable

• Accessible

• Versioned

Introduc)on*to*Lab*1Lab$1

User:&chefPassword:&chef

Time:&15&minutes

Welcome'Back• git%review%

git$pull

git$pull$REMOTE

git$push$remote$branch• transfer)commits)from)a)local)repo)to)a)remote)repo.

• counterpart)to)git)fetch

Infrastructure• Aggregate(of(applica/ons,(configura/ons,(access(control,(data,(compute(nodes,(network,(storage,(processes,(and(people.

Infrastructure+Automa/on• Systems(that(reduce(the(burden(on(people(to(manage(services(and(increase(the(quality,(accuracy(and(precision(of(a(service(to(the(consumers(of(a(service

Infrastructure+Automa/on+Tools• Chef

• Puppet

• Ansible

• Salt

• CFEngine

Introduc)on*to*Chef

Resources• Ingredients+of+infrastructure

• Basic+building+blocks

Resource(Declara,on

RESOURCETYPE "RESOURCE_NAME" do PARAMETER PARAMETER_VALUEend

Example(Resource(Type(1(package

A"package"to"be"installed:

package "httpd" do action :installend

Example(Resource(Type(1(service

A"service"that"should"be"started:

service "httpd" do supports :restart => :true action [:enable, :start]end

Resources

A"resource"is"a"statement"of"policy"that:

• Describes)the)desired)state)for)an)element

• Specifies)a)resource)type888such)as)package,)template,)or)service

• Lists)addi;onal)details)(also)known)as)parameters),)as)necessary

• Are)grouped)into)recipes

Recipes• Collec'on)of)ordered)resources

• Combina'on)of)ruby)and)Chef)DSL

Cookbooks• Thema'c

• Collec'on-of-recipes-and-other-suppor'ng-files

Roles• Abstrac)on,describing,func)on,of,system

• Name

• Descrip)on

• Run,list,(ordered,list,of,recipes,and,roles)

Run$List• Ordered&list&of&recipes&and&roles

• Specific&to&a&node

Nodes• Machine)(virtual,)physical,)cloud)server,)or)other)device))that)is)managed)by)Chef

Environments• Abstrac)on,models,workflow

• Name

• Descrip)on

• Cookbook,version,pinning

Supermarket• Community*site*with*a*number*of*cookbooks

• Read*before*using*in*your*environment

Chef%DK• Chef&development&kit

• Includes&a&number&of&u8li8es&and&so9ware&to&facilitate&cookbook&crea8on

• Free&download&off&of&the&website

Berkshelf• Dependency)management

• Included)with)Chef)DK

Test%Kitchen• Included)with)Chef)DK

• Sandbox)automa8on

• Test)harness

Test%Kitchen• Execute(code(on(one(or(more(pla1orms

• Driver(plugins(suppor7ng(various(cloud(and(virtualiza7on(providers

.kitchen.yml• driver

• provisioner

• pla-orms

• suites

.kitchen.yml,driver• virtualiza*on-or-cloud-provider

Example:)vagrant,)docker

.kitchen.yml,provisioner• applica'on*to*configure*the*node

Example:)chef_zero

.kitchen.yml,pla/orms• target'opera*ng'systems

Example:)centos/6.5

.kitchen.yml,suites• target'configura-ons

Example:

name:default run_list: - recipe[apache::default] attributes:

Kitchen(commands((1/2)• kitchen)init

• kitchen)list

• kitchen)create

• kitchen)converge

Kitchen(commands((2/2)• kitchen)verify

• kitchen)destroy

• kitchen)test

Docker• Images

• Registries

• Containers

Time:&30&minutes

Managing&Risk• Test

• Small*frequent*releases

Lin$ng• Ensure(code(adheres(to(styles(and(conven2ons

• Weave(expecta2ons(into(development

• Encourages(collabora2on

Tes$ng• Documen)ng+objec)ves+and+intent

• Measuring+"done"

Code%Correctness• foodcri(c

• rubocop

Integra(on*Tests• ServerSpec

Rubocop• Ruby&linter

• Ruby&style&guide

• Included&with&ChefDK

Rubocop'Example$ rubocop cookbooks/COOKBOOK1 cookbooks/COOKBOOK2 cookbooks/COOKBOOK4

Reading(Rubocop(OutputInspecting 8 filesCWCWCCCC

• ."means"that"the"file"contains"no"issues

• C"means"a"issue"with"conven2on

• W"means"a"warning

• E"means"an"error

• F"means"an"fatal"error✨ @beerops(@sigje(#effec/vedevops✨ 105

Disabling)Rubocop)cops

Any$configura,on$in$.rubocop.yml$is$disabled.

To#disable#string#literals:

StringLiterals: Enabled: false

Foodcri'c• Chef&linter

• Chef&style&guide

• Included&with&ChefDK

Foodcri'c(Example

$ foodcritic cookbooks/setup

Reading(Foodcri-c(OutputFC008: Generated cookbook metadata needs updating: ./metadata.rb:2

ServerSpec• Tests&to&verify&servers&func0onality

• Resource&types

• Package,&service,&user,&and&many&others

• Integrates&with&Test&Kitchen

• h?p://serverspec.org

ServerSpec'Generic'Form

describe "<subject>" do it "<description>" do expect(thing).to eq result endend

ServerSpec'Poten,al'Tests• Is$the$service$running?

• Is$the$port$accessible?

• Is$the$expected$content$being$served?

ServerSpec'Example

describe 'apache' do it "is installed" do expect(package 'httpd').to be_installed end it "is running" do expect(service 'httpd').to be_running endend

Reading(ServerSpec(Outputapp::default httpd service is running

Finished in 0.26429 seconds (files took 0.7166 seconds to load)1 example, 0 failures

Time:&30&minutes

Time:&20&minutes

Test,&Monitor,&or&Diagnos/c2

1. Where'is'it'going'to'run?

2. When'is'it'going'to'run?

3. How'o4en'will'it'run?

4. Who'is'going'to'consume'the'result?

5. What'is'the'en;ty'going'to'do'with'it?

2"Lam,"Yvonne."'Sysadvent:"Day"5"6"How"To"Talk"About"Monitors,"Tests,"And"DiagnosCcs'."Sysadvent.blogspot.com."N.p.,"2014."Web."26"May"2015.

Measuring*Impact*and*Value*of*Change

Impact'of'Change

Impact'on'Availability

• Overall(site/app(availability

• Individual(service(availability

Availability)Monitoring

• Up$me:

• Pingdom,/Moni$s,/Uptrends,/etc

• Ver$cal/Line/Technology:

• Availability/a?er/deploys/changes

Even%nator

Service'Availability

• Nagios:)Service/level)monitoring)and)aler5ng

• Nagios/herald:)Alert)context

• OpsWeekly:)Historical)alert)data

Nagios

define command { command_name check_mongodb_query command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -q $ARG5$}

define service { use generic-service hostgroup_name Mongo Servers service_description Mongo Connect Check check_command check_mongodb!connect!27017!2!4}

define servicedependency{ host_name WWW1 service_description Apache Web Server dependent_host_name WWW1 dependent_service_description Main Web Site execution_failure_criteria n notification_failure_criteria w,u,c}

Nagios'herald

OpsWeekly

Impact'on'Quality

• Service(quality((SLAs)

• Visibility(of(quality

Statsd

>>> import statsd>>>>>> timer = statsd.Timer('MyApplication')>>>>>> timer.start()>>> # do something here>>> timer.stop('SomeTimer')

>>> import statsd>>>>>> counter = statsd.Counter('MyApplication')>>> # do something here>>> counter += 1

>>> import statsd>>>>>> average = statsd.Average('MyApplication', connection)>>> # do something here>>> average.send('SomeName', 'somekey:%d'.format(value))

Graphite

Value&of&Change

Value&of&Availability

• Be$er&for&customers

• Be$er&for&employees&(internal&services)

• Fewer&pages

Value&of&Quality

• Deploys)take)less)-me

• Also)be1er)for)customers

• More)visibility)into)issues

Retrospec)ve+!

Review• Recognizing*your*Devops*Narra4ve

• Applica4on*Deployment*Planning

• Infrastructure*as*code

• Introducing*repeatable,*testable*change

• Measuring*impact*and*value*of*change

Next%Steps• Manual,(Automa-on(to(Con-nuous("X"

• Be(the(storylistener(and(storyteller(in(your(org

• Effec-ve(Devops(available(in(Early(Release

Thank&you!&❤@sigje

@beerops

effective devops - velocity new york 2015

Technology

di usive transport enhanced by thermal velocity...

2014 state of devops findings! velocity conference

mastering disasters - velocity ignite 2013 new york

velocity ny 2016 - devops: who does what?

2012 velocity london: devops patterns distilled

velocity...

why you can't buy devops in a box [velocity '14]

devops days new york

cloud scale predictive devops automation using apache spark:...

embracing devops to improve velocity and security - … ·...

world seventh edition australia and new ... -...

accelerating velocity and customer value with agile and...

devops and machine learning - geekwire€¦ · devops...

openstack and devops - devops meetup

devops, platforms and devops platforms

dev programming ops devops success...devops success damon...

accelerating velocity and customer value with agile and...

2013 velocity devops metrics -- it's not just for webops any...

devops : integrating development and operations for the...

realizing your potential “velocity” - urbancode.com ›...