cloud operations bootcamp: culture - jesse robbins
DESCRIPTION
Cloud Operations Bootcamp: CultureTRANSCRIPT
![Page 2: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/2.jpg)
Today
2
![Page 3: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/3.jpg)
Today
‣Operations is Culture
2
![Page 4: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/4.jpg)
Today
‣Operations is Culture
‣ Failure Happens
2
![Page 5: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/5.jpg)
Today
‣Operations is Culture
‣ Failure Happens
‣ The OODA Loop
2
![Page 6: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/6.jpg)
Today
‣Operations is Culture
‣ Failure Happens
‣ The OODA Loop
‣Do Fire Drills
2
![Page 7: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/7.jpg)
Operations is Culture
3
![Page 8: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/8.jpg)
4
“You don’t choose the moment, the moment chooses you.
You only get to choose how prepared you are when it does.” -Fire Chief Mike Burtch
![Page 9: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/9.jpg)
Cloud Operations is the ability to consistently create and deploy reliable software to an
unreliable platform that scales horizontally.
5
http://radar.oreilly.com/2007/10/operations-is-a-competitive-ad.html
![Page 10: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/10.jpg)
6
“It’s not my code, it’s your machines!
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 11: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/11.jpg)
6
“It’s not my code, it’s your machines!
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 12: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/12.jpg)
Spock ScottyLittle bit weird
Sits closer to the bossThinks too hard
Pulls levers & turns knobsEasily excitedYells a lot in emergencies
6
“It’s not my code, it’s your machines!
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 13: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/13.jpg)
Copyright © 2010 Opscode, Inc - All Rights Reserved 7
No "ngerpointing
http://www.!ickr.com/photos/rocketjim54/2955889085/http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 14: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/14.jpg)
Fingerpointyness
problem!!!argggh!
time
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 15: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/15.jpg)
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,finding fault
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 16: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/16.jpg)
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,finding fault
blaming,covering
ass
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 17: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/17.jpg)
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,finding fault
blaming,covering
ass
whining,hiding.
hurt egos
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 18: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/18.jpg)
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,finding fault
blaming,covering
ass
whining,hiding.
hurt egos
figuring it out
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 19: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/19.jpg)
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,finding fault
blaming,covering
ass
fixin
g th
ings
fixed
whining,hiding.
hurt egos
figuring it out
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 20: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/20.jpg)
Being productive
problem!!!argggh!
time
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 21: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/21.jpg)
Being productive
problem!!!argggh!
time
figuring it out
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 22: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/22.jpg)
Being productive
problem!!!argggh!
time
fixin
g th
ings
fixed
figuring it out
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 23: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/23.jpg)
Being productive
problem!!!argggh!
time
fixin
g th
ings
fixed
feeling guilty
figuring it out
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 24: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/24.jpg)
Being productive
problem!!!argggh!
time
fixin
g th
ings
fixed
feeling guilty
figuring it out
move on with
life
http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
![Page 25: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/25.jpg)
This will be on the test:FAILURE HAPPENS!
![Page 26: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/26.jpg)
Good Book!
![Page 27: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/27.jpg)
12
Complexity Complex
Loos
eTi
ght
Cou
plin
g
Simple
"Catastrophic Potential" adapted from Normal Accidents by Charles Perrow
Catastrophic Potential
Created by Jesse Robbins
![Page 28: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/28.jpg)
12
Complexity Complex
Loos
eTi
ght
Cou
plin
g
Simple
"Catastrophic Potential" adapted from Normal Accidents by Charles Perrow
Catastrophic Potential
Created by Jesse Robbins
KEEP OUT!!!
![Page 29: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/29.jpg)
define:Nines (roughly)
![Page 30: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/30.jpg)
define:Nines (roughly)
99% 5256 min (3.5 days)
![Page 31: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/31.jpg)
define:Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
![Page 32: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/32.jpg)
define:Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
![Page 33: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/33.jpg)
define:Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
![Page 34: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/34.jpg)
define:Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
99.9999% 30 Seconds
![Page 35: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/35.jpg)
define:Nines (roughly)
99% 5256 min (3.5 days)
99.9% 528 min ( 8.8 hours )
99.99% 53 min
99.999% 5 min
99.9999% 30 Seconds
99.99999% 3 Seconds
![Page 36: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/36.jpg)
99.9% *99.9% *99.9%
= 99.7%
14
![Page 37: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/37.jpg)
Internet Routing... won’t.
![Page 38: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/38.jpg)
![Page 39: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/39.jpg)
![Page 40: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/40.jpg)
![Page 41: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/41.jpg)
![Page 42: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/42.jpg)
!"#$$%"&'(')*)"+,-.,-/01,( +/.01210*"345467"89: #
;''-1(<"=/-)"3.1>0?-'"@'-':
![Page 43: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/43.jpg)
http://radar.oreilly.com/2008/10/sprint-blocking-cogent-network.html
![Page 44: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/44.jpg)
#googlefail
![Page 45: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/45.jpg)
Copyright © 2010 Opscode, Inc - All Rights Reserved
YOU
21
![Page 46: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/46.jpg)
Continuous Power... isn’t
![Page 47: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/47.jpg)
365 Main SF
![Page 48: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/48.jpg)
365 364.96 Main SF
![Page 49: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/49.jpg)
http://radar.oreilly.com/2007/07/failure-happens-a-summary-of-t.html
![Page 50: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/50.jpg)
http://radar.oreilly.com/2007/07/failure-happens-a-summary-of-t.html
![Page 51: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/51.jpg)
Failure happens
A single datacenter is the problem• Since they all fail at some point
Recovery procedures after failure• Power was gone ~45 minutes• Most services took hours to come back• Some unnamed ones more than 12 hours
![Page 52: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/52.jpg)
Geography is a Single Point of Failure
![Page 53: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/53.jpg)
![Page 54: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/54.jpg)
Copyright © 2010 Opscode, Inc - All Rights Reserved 30
![Page 55: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/55.jpg)
Providers are baskets too.
![Page 56: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/56.jpg)
Copyright © 2010 Opscode, Inc - All Rights Reserved 32
![Page 57: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/57.jpg)
Failure Happens.
Anyone promising otherwise is either foolish or lying
(or both).
![Page 58: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/58.jpg)
OODAObserve, Orient, Decide, Act
34
![Page 59: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/59.jpg)
35
OODA: Observe, Orient, Decide, Act
http://en.wikipedia.org/wiki/OODA_loop
![Page 60: Cloud Operations Bootcamp: Culture - Jesse Robbins](https://reader033.vdocuments.site/reader033/viewer/2022060108/554f4b5bb4c905524c8b4998/html5/thumbnails/60.jpg)
http://www.flickr.com/photos/dnorman/2678090600http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr