comparable simple actionable honest
TRANSCRIPT
From Vanity to Value: Metrics that Matter – Improving Lean and Agile, Kanban and ScrumDEV-B335
Steven BorgCo-Founder and Strategist, Northwest [email protected] @stevenborg
DEV-B335
Housekeeping
Steven Borg, Northwest [email protected]@stevenborg
Please don’t forget your evaluation!
Poorly thought through metrics
Not catching your flight home Airlines in the US
Hard hiking tripsPlanned economies
Losing your phone connection to customer service
Amazon
Getting great tech support… until bankruptcyGateway
Today’s Outline
Why Metrics“A Giant List of Metrics”Characteristics of a good metricVery Brief introduction to Kanban and ScrumMetrics for Scrum and KanbanOne Metric to Rule them AllCase Study
Metrics: The Bottom Line
Managers who don’t know how to measure what they want, settle for wanting what they can measure.
Russel Ackoff
What You See Is All There Is
The WYSIATI problem
Utilization vs Productivity, Capacity vs Flow
“Capacity: How must stuff will fit.Throughput: How much stuff will flow.
They are not synonymous.”- Jim Benson
Giant List of Metrics
Some metricsDefect Metrics
Bugs per KNCSS / KLOC / Function Point / User StoryBug rates for release-to-releaseMTTR – Mean Time to RepairAverage fixed defects/working dayAverage reported defects/working dayDefects/testing timePost-release defect rates (and arrival pattern analysis)Number or percent of defective fixesDefect severityDefect cause and problem component analysisCompile failures and build/integration defectsNumber of system crashes sand hangs during stress and system testing
Some more metricsProject Metrics
Schedule Estimation Accuracy Effort Estimation Accuracy NCSS/developer month or LOC/developer monthPercent overtimeUtilization RateNumber of developersDev/Test ratioTotal CostCost per KLOC / Function Point / User Story / Story PointAverage engineering hours/fixed defectPercent completeEarned Value (Completed vs Committed)
Some more metricsCode Metrics
ComplexityCode coverage of unit testsCode coverage of manual testsCode churn over timeRequirement changesSize in function points / KLOC / User Stories / etc
Reliability Metrics# of failures per n hours of operationMTTFProbability of failure-free operations in a specified timeNumber of undiscovered bugs released to production
Some more metricsCustomer metrics
Bugs reported per user month% satisfied, % dissatisfiedOverall customer satisfaction Customer problem calls per month
Process MetricsEffectiveness of bug removalResponse time to fixBug density during testing phaseBug arrival pattern during testing phasePhase-based bug removal patternCode coverageRequirement Changes
Defining metrics that matterWhat makes a metric effective?
Characteristics of Effective Metrics
Comparable
Simple
Actionable
Honest
Honest?Before
For i=1 To 10 Do Print i.ToString()Next i
After
Print 1.ToString()Print 2.ToString()Print 3.ToString()Print 4.ToString()Print 5.ToString()Print 6.ToString()Print 7.ToString()Print 8.ToString()Print 9.ToString()Print 10.ToString()
3x more productive?
True Metrics vs Proxy Metrics
True MetricWhat you SHOULD measure
Proxy MetricWhat you CAN measure
Looking at a few metricsCategorizing a few measures
Quality Metrics
Defect Removal EfficiencyWarning: Not always “simple”
Code CoverageWarning: Not always “honest”
Performance ProfilingWarning: Not always “actionable”
Test Cases per featureBugs per feature (bug density metrics)
Sizing Metrics
Lines of CodeWarning: Not always “honest”
Function PointsWarning: Not “simple”
Effort (in hours, days, weeks, etc)Warning: Not always “honest” or “comparable”
Story PointsWarning: Rarely “comparable”
Productivity Metrics
VelocityWarning: Not always “simple”
ThroughputLines of Code / Developer / Day
Warning on so many different levels
Quality*
Note: Defect Removal Efficiency is highly correlated with Productivity
So highly correlated it can often act as proxy metric for productivity
User Metrics
User satisfactionWarning: Not always “comparable”
Post-release bug countNumber of Help Desk callsWarning: Not always “honest”
Business Metrics
Schedule Estimation AccuracyWarning: Not always “honest”
Cost Estimation AccuracyWarning: Not always “honest”
Scope Estimation AccuracyWarning: Rarely “honest”
ROI Estimation AccuracyWarning: Rarely “simple”, Not always “honest”
Scrum and KanbanBrief (very, very brief) introduction
What Is Scrum?
Scrum Framework
Product Backlog
SprintBacklog
Sprint
1 – 4 weeks
24 hrs.
Daily Scrum
Working
software
What Is Kanban?
Kanban Basics
Visualize workMake policies explicitLimit work in process (WIP)Focus on flowCommit to continuous improvement
Scrum Kanban
Other Differences
ScrumTimeboxesOrdered backlogEstimationForecastingRemaining workVelocityCross-functional teams *Items sized to fit in sprintsWIP limited by sprintAdd to sprint by agreementScrum board reset each sprint
KanbanCadencesOrdering optionalEstimation optionalForecasting optionalCumulative flowLead timeSpecialists allowedNo item size requirementsWIP limited by stateAdd whenever capacity permitsKanban board persistent
Metrics for Scrum & KanbanAnd why they’re different that traditional metrics
Metrics for Scrum
VelocityBurndownWork CapacityCustomer AcceptanceAccuracy of Task EstimationAccuracy of Sprint CommitmentBusiness RevenueImpediment Impact per SprintDeveloper BurnoutTechnical Debt
Metrics for Kanban
Work In Process / Progress (WIP)Lead TimeCycle Time and Cycle Time DistributionThoughputBusiness RevenueFailure Demand Technical Debt
Lean and Agile Metrics Are Simple Focus is on fast software delivery
Feedback is king
Target high risks Market Risk vs Technical Risk
One (True) Metric to Rule Them All?
Time to Feedback
Feedback Loops
Measure
Adjust
Feedback Loops
Measure
Adjust
Five Steps to Any Process
Queue
Setup
Run Wait Move
Problems with Slow Feedback
Introduce more work into the systemLong time between code and bug fix makes it harder for developerLong time between bug find and bug fix verification makes it harder for the testerAdded complexityMore context switching
Lowers overall productivityBuild the wrong thing
Warnings
"When a metric becomes a target, it ceases to be a good
measure."
Goodhart's Law
Recall: True Metrics vs Proxy Metrics
True MetricWhat you SHOULD measure
Proxy MetricWhat you CAN measure
Time to Feedback Proxy MetricsMTTR4 (Realize, Recover, Repair, Remediate)Work in ProcessQueue lengthLead time / Cycle timeCode CoverageMean time to customer feedbackTouch time vs Lead time
Time to Feedback ToolsExperimentsA/B testingCustomer interviewsPrototypes and PretotypesContinuous delivery (eg. canary releases, feature flags, etc)Unit TestingATDD and TDDCross functional teams
Thought Exercise: Speeding Time to Feedback
Steven Borg@stevenborg
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)Work In
Process
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
WIP
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
WIPAverage Lead Time
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
6 items
WIPAverage Lead Time
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
6 items
Average Lead TimeWIP
Executive Decision to Limit WIP
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
WIP’
3 items
Speeding Time to Feedback
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
WIP’
3 items
Average Lead Time’
To deliver value more frequently, reduce work in
progress
Tip!
Little’s Law – “Two Minutes of Math”
Time
Am
ou
nt
of
Work
Assigned Work (over time)
Completed Work (over time)
WIP
Delivery Rate
Average Lead Time
Little’s Law – “Two Minutes of Math”
Delivery Rate = Work-in-Progress ÷ Lead Time
implies
Lead Time = Work-in-Progress ÷ Delivery Rate
thus
WIP is a LEADING metric for Lead Time
Manage queues, they are a leading indicator of upcoming problems
Tip!
Thought Exercise: Slack reduces impact of uncertaintySteven Borg@stevenborg
M / M / 1/ ∞
Lead
Tim
e
Utilization % 100%
80%
Lower utilization can dramatically improve lead time and flow of value.
Use WIP limits to constrain queue size in order to get
to feedback faster
Tip!
Case Study: Using a single metric to get faster time to feedbackSteven Borg@stevenborg
Touch Time vs Lead Time
Håkan Forss @VisualWIPvisualWIP.codeplex.com
Story Status:Working: 19Waiting: 31Delivered:
10
Håkan Forss @VisualWIPvisualWIP.codeplex.com
Story Status:Working: 38Waiting: 12Delivered:
10
Håkan Forss @VisualWIPvisualWIP.codeplex.com
Story Status:Working: 13Waiting: 4Delivered:
Lots
Faster Feedback with No Additional Work
January February March April May0
10
20
30
40
50
60
8.472
11.862 11.237 10.378
14.787
Hours of Throughput Hours of Effort Efficiency?
Observations:Multitasking reduced dramatically (over 50%)Work in progress reduced by 30-40%Work waiting for sign-off reduced by over 80%Throughput of value increased dramatically
Demo: Looking at Lead Times for Development TeamsSteven Borg@stevenborg
It’s your turn!
Steven Borg, Northwest [email protected]@stevenborgAnd please don’t forget your evaluation!
Questions?
DEV-B338 Better Together: Using Team Foundation Server and Visual Studio Online to Increase Agility
DEV-B206 Application Insights Overview: How to Keep Your Applications Available, Performing, and Succeeding
DEV-B317 Make Data-Driven Improvements to Your Application with Application Insights
DEV-B327 Deep Dive into Agile Planning for TFS 2013 and Visual Studio Online
DEV-B214 But, Is It Safe? A Closer Look at Visual Studio Online
DEV-B333 Cross-Platform Continuous Delivery with Release Management
DEV-B209 Best Practices for Using Open Source Software in the Enterprise
DEV-B334 Using Git with Team Foundation Server or Visual Studio Online
Related content
Connect with peers: DevOps meet up on Thurs (30th) 14:30–15:30 @ IT Community Experts in the Resource Zone (Expo Hall 7)
DevOps sessions @ TEE http://aka.ms/techeddevops
Resources for Devshttp://aka.ms/teched-eu
Resources for IT Opshttp://aka.ms/devopstl
Join the DevOps Insiders Group [email protected]
DevOps Resources
http://www.visualstudio.com
http://blogs.msdn.com/b/developer-tools/
http://msdn.microsoft.com/vstudio
DEV Track Resources
visualstudio
@visualstudio
visualstudio
Claim your Visual Studio Online domainMSDN subscribers, activate your Azure benefits now
Simply get started @ http://aka.ms/teched-eu
Resources
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Developer Network
http://developer.microsoft.com
Please Complete An Evaluation FormYour input is important!TechEd Schedule Builder CommNet station or PC
TechEd Mobile appPhone or Tablet
QR code
Evaluate this session
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.