an agile approach to business metrics
TRANSCRIPT
Product Owner’s Adventures in
Dataland
Who is this guy?@[email protected]
‘Conventional’ Agile Metrics
Lead Time*Defect count Work in ProgressCode coverageUnplanned ChangesVelocity (Stpts per sprint)Return on investment*Innovations per sprintArtifacts generatedSlack timeFailure Load (firefighting time)
Iteration Burn-DownUnfinished StoriesCustomer Satisfaction*LOC (lines of code)Un-deployed Stories# BlocksBudget/Schedule ComplianceFlow Efficiency (lead time / touch time)Release Burn-Up
*Italics: Connected to business value.
Your average Sprint Review...
KPIKPI KPI
KPI
Stakeholder
Scrum MasterProduct Owner
Is this ‘Agile’ enough?
KPIKPI KPI
KPI
Stakeholder
Scrum MasterProduct Owner
● Does it help increase customer satisfaction?● We deliver business value on every iteration,
but how do we measure it?● Shouldn’t the definition of metrics be an
iterative+feedback-based process on itself?● Priorities and markets change all the time.
Enterprise Metrics: The Ocean of Data
The Startup Way: Go Pirate
‘AARRR!’-Dave McLure500 Startups
The Lean Metrics Process (Croll & Yoskovitz)
The Lean Sprint (Ash Maurya)
The Thesis: The Analytics Sprint
Agile discipline and ‘Lean Analytics’ thinking can provide a better method to measure value.
The Analytics Sprint
Planning:
At the beginning of each sprint discuss: ● With stakeholders: What are the OKRs/key metrics? What’s the
KPI we should work on the next iteration? What’s the baseline?● With team: Hypothesis are defined on why/how to tackle that
KPI.
All of this: KPI, baseline, hypothesis and experiments are documented in writing.
The Analytics Sprint
Reviewing:
At the end of each sprint: ● There’s an assessment. Did we hit the goal? If not pivot into
another experiment. If we did we persevere onto the next KPI.● Data is turned into validated learning. Even failed experiments
provide learning (think Google with OKR).
The Analytics Sprint: The Metrics Master
Metrics Master: Role responsible for the definition, revision and accounting of the key metrics.● Facilitates discussion with stakeholders on
business value and key metrics.● Discusses hypothesis and experiments with
team.● Tracks and does innovation accounting.● Reports back on the results and either pivots or
perseveres on the next metric.● It can be the Product Owner (business value) or
a dedicated team-member (e.g. data-scientist)..
But wait, I already do that!Even if you’re already measuring a business value metric every sprint: ● When was its value defined?● Who defined it?● How’s the reporting being done to
stakeholders and customers? (Theatre of success or. customer discovery?).
Case Study: Spotify Metrics-Driven Development
Every retrospective they measure squad performance, and they refine the metric.
They also set goal targets at the start of every integration project we do (which may span multiple sprints). For example: “we will know we’re successful when this integration brings us x-amount of users.”
Source: Lynn Root
Case Study: Spotify Metrics-Driven Development
● They consciously avoid waste by measuring everything.
● They also set baselines with historical data.
● Goals are a shorter decision-making cycle as well as make more informed decisions about strategy and partnerships.
Some pro-tips
● Write it down. Prevents biases.
● Always define a baseline or benchmark.
● Avoid ‘bad metrics’.● Results will be murky.● Conversation with
stakeholders, customers and team is a value on itself.
Further Reading:Spotify: Metrics Driven Development by Lynn Root.Lean Analytics by Alistair Croll & Ben Yoskovitz.Appropriate Agile Measurement by Deborah Hartmann & Robin Dymond.Five Tips For Testing Your Idea by Anita Newton, Cindy Alvarez & Alistair Croll.Scaling Lean by Ash Maurya.Managing Metrics in an Iterative Incremental Development Environment by John D. McGregor.Agile Metrics, post by André Dhont.
A call to action:We like thinking of Agility not as a cargo cult, but as a set of values and principles that we adapt to the specifics of every single case.
So, we need you to learn.Please, hit me with your ideas, case studies and questions at [email protected]
Reasons for an Agile approach● We don’t know what we should be measuring.● We discover a better way to measure it.● Business priorities and political priorities change.● The product’s incremental development requires an incremental
an adaptative approach to its metrics.● Project (or transition) is too big for our resources and we have
to pick our battles.
We want to measure true business value as early as possible!
Challenges with this approach
From a business point of view:
● First, what information is available early in the process and how reliable is that information?
● Second, how can we use the early estimates to predict the actual values?
● The incremental aspect of the process requires that there be a plan for combining the values of certain metrics in order to obtain project-wide values (e.g. Addition, Averaging, Min-Max values).
Startup Vs. Intrapreneurial metrics (by A. Croll)