right - yow! conferences...release anti-patterns making easy = right 15 extensive, detailed release...
TRANSCRIPT
DevOps @ Wotif Making Easy = Right
Alexandra Spillane • @ajbw Matt Callanan • @mcallana
Right
Easy
Outline
• What is “The Right Thing”? • How do we Make it Easy? • Lessons Learned
Making Easy = Right 2
What is “The Right Thing”? DevOps @ Wotif: Making Easy = Right
Making Easy = Right 3
“The Right Thing”
Making Easy = Right 4
Collaborate!
Automate!
But… The Dev<->Ops Chasm
Making Easy = Right 5
What Were we Doing Wrong?
Making Easy = Right 6
Downward Spiral of Manual Activity
Making Easy = Right 7
Entrenched Silos
Making Easy = Right 8
High Release Overheads
Making Easy = Right 9
Huge Batch Sizes
Making Easy = Right 10
Calendar of Doom
Making Easy = Right 11
Unpredictable Prioritisation
Making Easy = Right 12
Superstitious Gatekeeping
Making Easy = Right 13
just in case
Stagnating Applications
Making Easy = Right 14
Release Anti-Patterns
Making Easy = Right 15
Extensive, detailed release documentation
Reliance on manual testing to confirm app is correct
Explaining why deployment is going wrong on release day
Frequent corrections to release process during release
Environments that differ in their configuration
Releases that take more than a few minutes to perform
Releases that are unpredictable in their outcome
Cycle Time measured in
weeks/months
Application Pipelines Ops
Days
Weeks!
Days
Staging Load Test Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA !?
Prio
ritis
ed R
elea
se Q
ueue
Plus… Migrating to MSA
• Explosion of manual effort
Making Easy = Right 17
A B
C D
E F
A
B1 C1 G C2 H I
B2 D1 J
D2 K L1 M L2 N
E F1 O P
F2 Q R S T
HeavyWeight to LightWeight
Glassfish • Feature overhead • Encouraged manual config • Complex/Slow deployment
DropWizard • Trimmed down to basics • Standard config files • Simple/Fast deployment
Making Easy = Right 18
But… Combinatorial Explosion
Making Easy = Right 19
Unpredictability Slows You Down
Making Easy = Right 20
Moving from wrong to right
Making Easy = Right 21
Wrong
Right
Reduce Cycle Time
Making Easy = Right
Standardisation
Making Easy = Right 23
Standardised
Freedom
• Standardise on how services are deployed and how they communicate.
• Be flexible about their contents.
Defining Standards
• Discussions – One-on-one – IM – Emails – Meetings
• Incentivise input • Confluence
– Application Deployment Standards 1.0
Making Easy = Right 24
Example Standards
Making Easy = Right 25
Logging • Log file locations, filenames
Log rotate Directories/Files • Ownership,
permissions
Puppet
Supervisord config.yaml • Filename, location
Metrics JVM settings
Network • Ports • Firewall rules
Cron Endpoints • Healthchecks, status,
metrics
SSL • Certs, keystores
RPM • Versioning • Packaging
init.d scripts • Sub-commands
Versioning Migration Notes
Example Standards
Making Easy = Right 26
Future Considerations
Making Easy = Right 27
Standards Lifecycle
Making Easy = Right 28
Common Ground
Making Easy = Right 29
How do we Make it Easy? DevOps @ Wotif: Making Easy = Right
Making Easy = Right 30
Automated Verification
Making Easy = Right
Compliance Test Suite Bring Operations into Development Fast Operational Feedback Backwards Compatibility Test Driven Operational Compatibility
Ops Testing with Fabric
Making Easy = Right 32
• Does rpm have correct metadata linking to git repo? Check existence/absence of files rpm • Is correct version actually installed? yum • Check existence/absence of files. Check file permissions/ownership ls • Is correct version actually running? lsof • Check correct ports exposed netstat • Is correct log format in use? tail • Check cron config grep • Check contents of standalone jar file – e.g. metadata, library versions unzip • Check standard endpoints for e.g. content, status code, response time curl • Check JVM options jps • Check 1 and only 1 process running ps
Compliance Test Example
Making Easy = Right 33
Reference Implementation
• “helloworld-service” • Deployed to production
Making Easy = Right 34
Rolling Upgrade
• Safely automate cluster upgrades • Orchestrated with Fabric
Making Easy = Right 35
• Check Load Balancer
Exit Pool
• Orchestrate Puppet
Upgrade
• Compliance Test • Smoke Test • Feature Test
Test
• Check Load Balancer
Enter Pool
• exitpool • stop • yum remove • yum install • Wait for manual
testing! • enterpool
Manual Release
SLIPway
Making Easy = Right 36
• Big review of old release processes
• Simple Lightweight Independent Path Way
• Simple rules for release process
• Keep changes independent
• Focus on reducing cycle time
Simple Rules for SLIPWay
• Independent • One application per release • Backwards-compatible • No DB/Network/OS changes
• No manual testing • Cannot book specific time
• Feature switch for time • Ops will service queue within 24 hours
• Compliance with latest standards
Application Pipelines Ops
Days
Weeks!
Days
Staging Load Test Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA !?
Prio
ritis
ed R
elea
se Q
ueue
39
Application Pipelines Ops
Prio
ritis
ed R
elea
se Q
ueue
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Hours Hours
Sim
ple
Rule
s
Making Easy = Right 40
SLIPway Chat Room
Making Easy = Right 41
SLIPway Example Release
Making Easy = Right 42
Freeing up 3 full time employees
0
5
10
15
20
25
Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
Man
-hou
rs p
er re
leas
e*
man
-hou
rs t
o sh
ip a
rel
ease
95%
reduction in man-hours
GRIPway
SLIPway
SLIPway introduction
-18.5hrs
Hours Spent Shipping Release
0
5
10
15
20
Cycl
e ti
me
in b
usin
ess
days
busi
ness
day
s -8
-2
86% cycle time reduction
GRI
Pway
SLIPway
GRIPway average 2013
GRIPway today
Jan Jan Jul Apr Oct Apr
2014 2013
Jul Oct
Time-to-market Reduction
Making Easy = Right
0
20
40
60
80
100
120
Num
ber
of r
elea
ses
per
mon
th
Thro
ughp
ut x
2.9
190% releases
GRIPway average 2013
SLIPway introduction
GRI
Pway
SL
IPw
ay SLIPway average
GRI
Pway
Jan Jan Jul Apr Oct Apr
2014 2013
Jul
2.9x Releases Per Month
Making Easy = Right
Release Cycle Time Comparison
Making Easy = Right
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days
Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days
Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days
Commit stage
Acceptance stage
Capacity Testing
Production
Con%nuous Delivery
Automa%c Trigger Manual Trigger
Commit stage
Acceptance stage
Capacity Testing
Production
Con%nuous Delivery
Con%nuous Deployment
Automa%c Trigger Manual Trigger
Commit stage
Acceptance stage
Capacity Testing
Production
Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time
21mins
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT
Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time 30x lower vs SLIPway 200x lower vs GRIPway
21mins
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT
Lessons Learned DevOps @ Wotif: Making Easy = Right
53
Lessons Learned
Making Easy = Right 54
Get the Balance Right Chip Away At The Iceberg The Carrot, Not The Stick Strongly Type Your Discussions Use Semantic Versioning Embrace Legacy
Get the Balance Right
No Standards Too Many Checkpoints
Making Easy = Right 55
Chip Away at the Iceberg
Making Easy = Right 56
The Carrot, Not The Stick
• We tried the stick approach for years – You’re doing it wrong! – (But we won’t tell you how
to do it right)
• Blame games, anger • No business impetus to
improve things
• Now we’re trying the carrot approach – Offer a great new
alternative – It’s optional! – You know you want it…
• Business invested in the process
Making Easy = Right 57
Strongly Type Your Discussions
• Choose catchy names – Helps make decisions – Helps climb out of muck
• Use State Transition tables – State -> Action -> New State – Visualise problems – Decide what to support
Making Easy = Right 58
E.g. Glassfish Service States
Making Easy = Right 59
E.g. Lightweight Service States
Making Easy = Right 60
Use Semantic Versioning
• Especially for shared libraries
Making Easy = Right 61
1 . 2 . 3 Major Minor Patch
semver.org
Be Opinionated – Not Arrogant
• Strong decisions are important • But you’ll never get it 100% up front
Making Easy = Right 62
I’m kind of a big
deal
Building Blocks SLIPway • Simple Rules, Independence
Rolling Upgrade • Automated Deploy & Test
Compliance Tests • Test-driven Ops Compatibility
Standards • The Agreed “Right Thing”
Making Easy = Right 63
The Right Thing
Making It Easy
Making Easy = Right 64
Right
Easy
Making Easy = Right 65
Right
Easy
Thanks for listening! Any questions?
Alexandra Spillane • @ajbw Matt Callanan • @mcallana
slides, etc • mattcallanan.net
Making Easy = Right
Image Attribution • Creative Commons:
– Downward Spiral: http://flic.kr/p/67Giiz – Chasm: https://flic.kr/p/abuWHt – Wrong way: https://flic.kr/p/cRjc6q – Silos: https://flic.kr/p/6WueVm – Testing Ice-cream: http://watirmelon.files.wordpress.com/2012/01/softwaretestingicecreamconeantipattern.png – Lifting boat: https://flic.kr/p/ddUf4S – Batch Size: http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/ – Changed Priorities: https://flic.kr/p/debvm – Stop: https://flic.kr/p/aUEW1D – Stagnant: https://flic.kr/p/3MvvsH – Thinking: https://flic.kr/p/6wdLat – Rubber Bands: https://flic.kr/p/a2XZDB – Climbing: https://flic.kr/p/4U2BB1 – Cycling: http://commons.wikimedia.org/wiki/File:Alexander_Vinokourov_2,_London_2012_Time_Trial_-_Aug_2012.jpg – Indian Traffic: http://frrl.files.wordpress.com/2012/07/logo_indiatraffic.png – Red Tape: https://flic.kr/p/b7T3JD – Slipway: https://flic.kr/p/9oQVh8 – Iceberg: https://flic.kr/p/hV1vwH – River: https://flic.kr/p/7EMs4T
• Additional Stock photos: – iPhotoStock – Shutterstock
• Standardisation Theory – Diagram inspired by: http://www.slideshare.net/spnewman/practical-microservices-yow-2013/56
Making Easy = Right 67