Reduce Time to ValueFocus First on Configuration Management Debt
Chris Sterling, Product Owner, CenturyLink Cloud
Bio - Chris Sterling
2
Author of “Managing Software Debt: Building for Inevitable Change” Supported change efforts to adopt Lean, Agile & Continuous Delivery behaviors for organizations of 10 up to 800+ people Product Owner at CenturyLink Cloud Entrepreneur & Lean Startup Practitioner Blog: managingsoftwaredebt.com
Social
3
@csterwa #swdebt
Types of Software Debt Configuration Management Debt: Integration and release management become more risky, complex, and error-prone.
Platform Experience Debt: The availability and alignment of people to business objectives that involve software changes is becoming more limited or cost-prohibitive.
Design Debt: The cost of adding features is increasing toward the point where it is more than the cost of writing from scratch.
Quality Debt: There is a diminishing ability to verify the functional and technical quality of software: the “Break/Fix” mentality.
Technical Debt: These are the activities that a team or team members take shortcuts on now that will impede future development if left as is.
4
Reducing Time To Value
5
Focusing on Configuration Management Debt first leads to opportunities for reducing all forms of software debt Reduce hand-offs and dependencies in Org to reduce Platform Experience Debt Malleable architectures with Pluggable UI, Microservices and APIs to reduce Design Debt Increase efficiency of validation processes to reduce Quality Debt Make software more testable to reduce Technical Debt
Configuration Management
Debt
“If releases are like giving birth, then you must be doing something wrong.” — Robert Benefield, CTO - Evolve Beyond
6
Case Study: Web Property
7
180+ person “Web 2.0” product organization Waterfall SDLC that development uses to deliver in 6 month release cycles Want to use Agile methods to be more responsive to users and keep up with other “Web 2.0” companies Transitioned to Agile methods on 15 teams in 3 months Changed release management strategy, added XP technical practices, and implemented Scrum product development framework for scaled coordination Able to release every week to users within 4 months Used streamlined deployment environment process to validate product changes daily using Continuous Integration and automated promotions
Case Study: Ad Platform
8
700+ person Ad Platform organization
Extended dependencies out to 2000+ people throughout company
Millions of lines of code + Hundreds of unique apps/services
Continuous Integration server involved thousands of jobs & 1M+ builds per year
9-18 month overlapping release cycles
Found opportunity to not branch for each overlapping release
After 2 years can release to 20+ data centers globally every day
4 years later they release to production with no humans for nearly 100% of all their company apps/services
Principle:
Focusing on Configuration Management Debt first leads to opportunities for reducing all forms of software debt
9
Where Does CM Debt Source control practices that impact delivery velocity
Too many variations/versions of the software supported in production
Over-burdened release engineering and operations teams
High coupling with centrally managed architecture element/component
Too many hand-offs between teams in order to release software to users
Poor integration processes across architecture components and scaled team delivery
Code changes feel too risky and takes too long to validate before releasing into production
Poor documentation practices10
Traditional Source Control
11
Traditional Source Control
11
Main Branch
Traditional Source Control
11
Main Branch
Version 1 Branch
Integrate for Version 2
Code Complete
Traditional Source Control
11
Main BranchDebt
Death March
Version 1 Branch
Integrate for Version 2
Code Complete
Traditional Source Control
11
Main BranchDebt
Death March {Debt accrues quickly within stabilization periods
Version 1 Branch
Integrate for Version 2
Code Complete
Worse Source Control
12
Main Branch
Version 1
Version 2 Version 3 Version 4
Worse Source Control
12
Main Branch
Version 1
Version 2 Version 3 Version 4
{We wish branch points were this clean
Flexible Source Control
13
Flexible Source Control
13
Main Branch
Flexible Source Control
13
Main Branch
Version 1
Flexible Source Control
13
Main Branch
Version 1 Version 2
Flexible Source Control
13
Main Branch
Version 1 Version 2{Not Easy! Must have proper infrastructure to do this.
Scaled Continuous Delivery
14
Component Validation
Integrated Component Validation
End-to-End & Load/Stress
Canary Deployments
15
Embrace DevOps Culture
Cultures, for better or worse, are very stable. — Jeff Bezos, CEO Amazon
16
Principle: Teams own software delivery from solution to deployment to operations.
17
Monitor App Performance
18
Assess Anomalies
19
Monitor Valuable Trends
20
Platform Experience Debt
“As in Nature, if an organization is too inflexible or stands still too long it will get eaten.”
- James Burke
Principle: Rather than creating teams to work on projects, let’s find ways to give work to cross-functional teams.
22
Feature Team “Feature Team” structure Uses common Product Backlog Integration is done in parallel Requires high levels of communication across teams to resolve integration issues Forces Product Owners to be more coordinated Sprints should be synchronized Cross team fertilization is a requirement to successfully deliver in parallel
23
Design DebtTechnical features that involve improving
software quality attributes can be prioritized based on the cost of not addressing them.
25
Malleable architectures and design enable nimble business capabilities.
Feature Toggles
Problem: Need to deliver changes to production on master with features that are not sufficient or validated with customers
Introduce feature behind toggle ▶ ON: Show and allow access to feature ▶ OFF: Don’t show or allow access to feature
Should be secondary option to introducing smaller capabilities that lead to sufficient feature
26
Microservices
27
Pluggable UI
API Routing
Service 1 Service 2 Service 3 Service 4
Identity Management
Authorization Authorization Authorization Authorization
Data Data Data Data
Messaging Platform
Application Programming
28
API
Deploy to a Platform
29
Quality Debt“Promises make debt, and debt makes
promises.” — Dutch proverb
The Three Amigos
Quickly get testers, coders, and business on the same page before building based on a requirement
31
* The Three Amigos pattern originally coined by George Dinwiddie http://www.stickyminds.com/s.asp?F=S17232_COL_2
The Three Amigos Pattern
32
As a Shopper I want to receive updates on incredible deals that are located near my home so that I can save money on my purchases
Acceptance Criteria: Save Shopper’s location Ask Shopper if they want to receive localized deals daily Send notification of incredible deals to Shoppers located within 10 miles each morning Allow Shopper to stop receiving localized deals
The Three Amigos Pattern
33
As a Shopper I want to receive updates on incredible deals that are located near my home so that I can save money on my purchases
What areas of the application will this affect? What is the overall design? (UI, API, UX, etc…) What are the details test cases for this user story and it’s acceptance criteria? What about negative test conditions? What about boundary conditions? How might we put existing functionality at risk?
The Three Amigos At minimum include tester, coder & business rep in discussion Should only take 30 minutes to 1 hour for user stories Focus on clarification and design through testable inputs/outputs Extend to Operations to reduce CM debt
34
Acceptance Test-Driven
35
ATDD Case StudyTest Automation Reduces Cost of Change
Manual Regression Testing
Testing was taking 75 person hours during 2 full test runs consisting of: ▶ Comprehensive manual regression testing ▶ Data conversion and validation Cost for testing was $17,000 each iteration
37
Introducing Fit into Testing After 8 iterations team had introduced healthy amount of Fit fixtures and automated tests Reduced 70+ hour test runtime down to 6 hours which now included: ▶ Fit automated regression testing ▶ Data conversion and validation automated with Fit
fixtures Reduced cost of testing each iteration from $17,000 to $7,000
38
The Agile Regression Testing
39
* The Agile Triangle has been modified from Mike Cohn’s original version
The Agile Regression Testing
39
* The Agile Triangle has been modified from Mike Cohn’s original version
Automated Unit Tests Make up largest portion of regression tests and are developed by programmers
The Agile Regression Testing
39
* The Agile Triangle has been modified from Mike Cohn’s original version
Automated Unit Tests Make up largest portion of regression tests and are developed by programmers
Integration TestsAutomated &Exploratory
The Agile Regression Testing
39
* The Agile Triangle has been modified from Mike Cohn’s original version
Automated Unit Tests Make up largest portion of regression tests and are developed by programmers
Integration TestsAutomated &Exploratory
Smoke++ Tests Risk-based UI & API Automated Tests
Technical Debt
“For every [dollar] of competitive advantage gained by cutting quality, it costs $4 to restore it; and software is an organizational asset and decisions to cut quality must be made by executive
management and reflected in the financial statements.” — Ken Schwaber, co-creator of Scrum
Source: http://www.infoq.com/presentations/agile-quality-canary-coalmine
Continuous Integration
41
Early Warning Signs
42
Early Warnings: • Broken Builds • Broken Automated Tests • Broken Custom Thresholds
Quality Dashboard - Sonar
43
Early Warning Dashboard
44
Measuring Early Warning Signs: • Design Debt in Duplication (DRY) • Technical Debt in Code Complexity • Quality Debt in Bug DB (Break/Fix) • Other Custom Thresholds
Reducing Time To Value
45
Focusing on Configuration Management Debt first leads to opportunities for reducing all forms of software debt Reduce hand-offs and dependencies in Org to reduce Platform Experience Debt Malleable architectures with Pluggable UI, Microservices and APIs to reduce Design Debt Increase efficiency of validation processes to reduce Quality Debt Make software more testable to reduce Technical Debt
Q&A