does16 london - better faster cheaper .. how?
TRANSCRIPT
Better, Faster, and Cheaper. How?
Damon Edwards John Willis
Co-Founder & Managing Partner DTO Solutions, Inc.
Damon Edwards @damonedwards
Director of Ecosystem Development Docker, Inc.
John Willis @botchagalupe
devopscafe.org
30x 200xmore frequent deployments
faster lead times
60x 168xthe change success rate
faster mean time to recover (MTTR)
2x 50%more likely to exceed profitability, market share & productivity goals
higher market capitalization growth over 3 years*
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
Recent IT Performance Data is Compelling
30x 200xmore frequent deployments
faster lead times
60x 168xthe change success rate
faster mean time to recover (MTTR)
2x 50%more likely to exceed profitability, market share & productivity goals
higher market capitalization growth over 3 years*
High performers compared to their peers…
Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report
Recent IT Performance Data is Compelling
Faster
HigherQuality
MoreEffective
Fast
CheapGood
“Pick Two!”
Conventional Wisdom
Faster, Better, and Cheaper?
Faster, Better, and Cheaper. How?
Culture Automation Measurement Sharing
Culture Automation Measurement Sharing
Better, Faster, Cheaper. How?
Org structure that stays out of the way
Better, Faster, Cheaper. How?
Dev Release OpsPlanning
Application Knowledge
Operational Knowledge
Business Intent
Handoff!
Handoff!
Handoff!
Ownership but limited Accountability
Accountability but no Ownership
Silos are enemy of throughput and stability
Dev Release OpsPlanning
Focus on reducing handoffs and improving flow
Service Aligned Delivery Teams
Service Aligned Delivery Teams
Service Aligned Delivery Teams
Environments as a
Service
QA as a
Service
Insights as a
Service
Security as a
Service
Environments Team
QA Team
Monitoring & Metrics
TeamSecurity
Team
Org structure that stays out of the way Build everything through a SDLC
Better, Faster, Cheaper. How?
Average Peformers: SDLC for applications
Code
Source Repo
Config
CI Server
Package Repo
Operations Console
Application Deployment Automation
Pre-Production Environments
Production Environment
Packages
Ope
ratio
ns
Dev
elop
men
t
SOURCE
Tests
Application Deployment Automation
Code
Versioned Release
Source Repo
Config Env Spec
Run- book
Auto-mation
CI Server
Package Repo
Operations Console
Pre-Production Environments
Production Environment
Packages
Ope
ratio
ns
Dev
elop
men
t
SOURCE
Dev Ops *
Collaboration
Dev Ops *
Execute Operations Procedures
Same People!!
Tests
High Performers: SDLC for everything
Org structure that stays out of the way Build everything through a SDLC Make the work visible
Better, Faster, Cheaper. How?
Service now
Parts Unlimited - "Major Release 6"
Early 2014
Project Initiation
ZRA (finance)
Approve Project
Monthly Steering Meeting
Portfolio
C-level
Steering Comittee
Provides Input
Project Charter
High-Level• Stories• Project Info• Description• Budget• Schedule
PMStakeholders (Tech and Biz)
Create Work Breakdown
Work Breakdown (MS Proj)
High-Level• Milestones• Resource
Planning
3 months 3 monthsHold / Pause
Create Requirements
(Project Meeting)
MS Office
• Detailed Req for new features
• Technology refreshes
• ERD (Infra req)• DRD (Dev req)• BRD (Biz req)
Share Point
Create Design
Tech ReqTech
ReqTech Req
Tech Leads Architects Vendor Arch
Ops Arch
High-LevelServer Tickets
3 months
Receive Request for
Servers
Create Server
Request Spreadsheet
ServerReq
PMTixattach
Route for Approval
Tix
1 week 1 week
• Budget• Appropriate
Resources DB
App or Web
orApproved Into Ops
Delivery Queue
Delivery Manager
"Matt"
Service now
"Heads up"
Assign to Delivery Engineer
Delivery Engineer
Clarify or Confirm Req with Dev or
QA
1 - 6 weeks
Provision Server
and Rework
DBA Validation
App/Web Validation
RestoreData
1 weekApp
Team
App Team
PMStakeholders (Tech and Biz)
Dev Leads
4 weeks
ARB Queue
Detailed Analysis and Requirements
Jira "Stories"
Maybe
Track Ticket Dependencies
Confluence Pages
Team Leads and PMs
Assign Requirements
add more detail for their teams
Architecture Review Board
"Bill" plus Architects
Working Group
Ops? (sometimes)
Devs, PM, Engr, QA
Development Sprint
2 week c/t
Existing Dev Environments
Acquire / Prepare needed
dataOps DBA
Service Data Setup
(Mainframe)
"Jennifer"
Test Data Configuration
Manager
Development Deploy to Integration
Dev, QA
Integration & Regression
Testingfocused on service
ScrumDev/QA
Integ03
ScrumDev/QA
Test Link
Sprint Review
Release to Prod
Product Owners(Using own
criteria)
Create CAB ticket
or
Scrum Team Ops Team(if legacy)
Push Deployment to Stage
Stage
Email Notification
Jira
NewArch
Build VMs
Jira
Ops
ServiceNow
Legacy
QA LeadPMsQAs
End to end testing in Prod
Prod Env
PrdDB
Go-No Go decision meeting
Team Leads
Jira
Ops
By Cluster
"Remove Feature Flag"
(if new arch)
16 weeks
6 weeks H/C: 6 3 weeks H/C: 8
4 weeks H/C:8 3 weeks H/C: 14
Data Setup Integration Testing
DEv Arch
Create Change Tickets > 100
Service Now
ComputeNet
FacilityCablingStorage
"Linda"Ops PM
RESET DELIVERY
DATE!
Steering Comittee
Fix Tickets!
"Linda" Ops PM
Dev Leadership
Assign Dev Team
Ops Intake Meeting
Dev Leadership
1 week
GroupCIOs and
Arch Leads
QA
SteeringDesign
Dev BreakdownDev / Test
Staging Release
Server Requirements GatheringServer Approval and Assignment
Provisioning
Production Release
Initiation and Planning
Create OpsTickets
TS PD
TS PD
Gaps in Requirements• Licenses• Dependencies on 3rd party apps• Capacity planning always seem low
("robbing Peter to pay Paul")• Don't purchase in advance even though
we know it's coming
Duplicate info across different documents
EP
D
D
Procurement of physical servers can take months (lead times for procurement plus facilities groups)
Too many Env. in on ticket cases audit confusionPiecemeal requests ("2 this week, 3 next week")
1 queue for delivery team with ~1,000 tickets at once
Capacity issues cause delay
Often told to stop everything and do something else
TS
D
M
TS
M W
W TS EP
HNo monitoring or backup for some environments
30% of delivery teams time spent "consulting" on performance and dealing with unfounded requests for more capacity
3-5 days to fix~10% S/R
H
D M
TS
H
Often skips CAB. What CAB reviews is often not what built
All manual setup. 1 person really knows how. Low data quality.
Manual process with lots of back and forth.
Many tickets with mismatched priorities
Mostly manual testing
Manual, per clusterFrequently down.
External service updates take offline. Lots of contention.
EPM
D
PDM W
TS
TS D
M TS
PDM
M
S/R - 90%
S/R - 55%
S/R - 15%
D
S/R - 20%
S/R - 50%
Sometimes submits server requests directly to delivery Ad-hoc requests get
lost, maybe 2-3 week delays
TS
High Level
S/R - 75%
9+ months of planning before implementation starts
(and information / requirements still incorrect or incomplete!)
Dev and QA told to submit sever request 6-8 weeks in advance (only done 50% of time)
W5. New "white glove" engagement model
3. Standard product catalog("Environments on Demand")
2. Visualization of flow of work and expected upcoming work
4. Shorten from Design to Implementation
1. Fully Automated Environment Provisioning
7. Small Batches
8. Write end-to-end customer
func. tests
11. Resolve interface to
legacy
10. Test data setup
automation
13. Dev Deploy to Prod for legacy
14. Unify change
management tools
15. Tool
9. Service Verification test writing: shift left to Dev(test early)
12. Remove Bottleneck and Environment Contention(test more)
• Make the work visibile for all • Build alignment and consensus across team
boundaries • Empower teams to find and fix what is getting in the
way
Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery
Better, Faster, Cheaper. How?
• Variety• Determine your variety of offerings
based on operational efficiency and market demand
• Velocity• Maintain a steady flow through all
processes of the supply chain• Variability
• Manage inconsistencies carefully to reduce cost and improve quality
• Visibility• Ensure the transparency of all
processes to enable continuous learning and improvement
Immutable Delivery helps realize Toyota’s 4VL
Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery Microservices
Better, Faster, Cheaper. How?
Microservices
“Small Autonomous Services That Work Together”
-Sam Newman
“Loosely Coupled Service Oriented Architectures With
Bounded Context” -Adrian Cockroft
Working with Monlithic Services
Slide stolen from Adrian Cockcroft
Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release Integration
Ops Replace Old With New
Release
Monolithic service updates
Works well with a small number of developers and a single language like php, java or ruby
Working with Monlithic Services
Slide stolen from Adrian Cockcroft
Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release Integration
Ops Replace Old With New
Release
Bugs
Bugs
Monolithic service updates
Works well with a small number of developers and a single language like php, java or ruby
Working with Microservices
Slide stolen from Adrian Cockcroft
Developer
Developer
Developer
Developer
Developer
Old Release Still Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy Feature to Production
Deploy Feature to Production
Deploy Feature to Production
Deploy Feature to Production
Bugs
Deploy Feature to Production
Immutable microservice deployment scales, is faster with large teams and diverse platform components
Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery Microservices Respect for People
Better, Faster, Cheaper. How?
High-Performers care for their people like strategic assets
• Diversity • Empathy • Areas of Worklife Scale (AWS)
It’s not just humane, it’s a business advantage
Recipe for burnout is inverse of recipe for success
Work overload: Job demands exceed human limits.
Lack of control: Inability to influence decisions that affect your job.
Insufficient rewards: Insufficient financial, institutional or social rewards.
Breakdown of community: Unsupportive workplace environment.
Absence of fairness: Lack of fairness in decision-making processes.
Value conflicts: Mismatch in organizational values and the individual’s values.
Source: Dr. Christina Maslach’s six organizational risk factors that predict burnout (via Dr. Nicole Forsgren)
Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery Microservices Respect for People
Better, Faster, Cheaper. How?
Co-Founder & Managing Partner DTO Solutions, Inc.
Damon Edwards @damonedwards
Director of Ecosystem Development Docker, Inc.
John Willis @botchagalupe
devopscafe.org