measuring product management/ownership effectiveness in a lean/agile world

Metrics for the LEAN/Agile Product Manager

Yuval Yeret

Main things we want to pay attention to

• Performance of the Production floor – covered elsewhere (Simple KPIs Slides)

• Performance of the Product Management group: – Business Value–Wastes related to PM– Technical Debt

Business value

• We care about outcome – features delivered, adopted, used, paid for

• How can we measure this? • Manage a kanban at the high level features level,

that tracks when features are adopted, and upon first paying customer.

• Then see how much WIP of features not yet adopted we have, LEAd/Cycle time to adoption, features that we dropped on the way.

Debt

• A lot of time debt is taken due to PM decision

• We want to track how much debt we have, and take action to minimize it.

• E.g. we need to release ASN1 now, so we don’t “automate tests”/”code it correctly”, so every work on ASN1 is slower, until this is fixed

Tracking debt in kanban

• Have debt card type that is created when debt is taken on

• Track amount of debt versus overall WIP/Backlog

• See whether stable, improving, worsening trend

• Decide on policy for dealing with debt – WIP Limit, etc.

• Track the cycle time and WIP for debt cards to see whether they get the SLA they deserve

Wastes related to PM

• Waiting for PM• PM related Churn / Context switching /

Expediting• Sunk Costs• Rework due to late feedback by PM

Waiting for PM

• Look at the CFD, observe the size of the PM-related queues over time. – Especially Pending PM Review which is in the

middle of DEV/Test– And Ready-MMFs as well as DEV Ready in some

cases which depend on PM approval– Advanced – in the cycle time performance

report, focus on PM areas

• When looking at exceptions to Cycle time, participate in the root cause analysis, and see if interaction with PM was part of the long cycle time.

PM related Churn / Context switching / Expediting

• Add Expedited class of service– Can be used by PM to

override priorities in DEV WIP – just to top of queue, don’t override current WIP

• Add emergency class of service– Can also override current

WIP

• Assumption – – This is value trumps flow.

We give up efficiency when we use these COSs

Measuring the effect of value over flow COS

• Look at cycle times for different kinds of classes of service

• Look at distribution of different COS in the WIP

Look at amount of changes in scope

• Replace – need visualization that shows scope changes in content

• Add – can simply look at total scope for a “Release” and observe whether its growing

Case Study – Typical release behavior

1 2 3 4 5 60

20

40

60

80

100

120

140

160

180

200

Total Required WorkLinear (Total Required Work)Actual DonePlanned ScopePlanned DoneOriginal + Dark MatterLinear (Original + Dark Matter)

Added Scope

Growth in Feature

Cost / Dark Matter

• Dark Matter – Is where we thought a feature costs X

• But then, during breakdown, analysis, creating iteration stories, we understand it actually costs X+D

• Then, PM decides whether to scope to fit down to X again, or D is worth it.

• Worthwhile tracking our behaviour on this, and learning from it.

• What is the right D number/percent? Good question!

• Can be observed in the CFD for a release.

Sunk Costs

• Add a LANE that collects features/stories that are “ON HOLD” – the Recycle Bin in the archive area

• The amount of work done on them is the sunk cost

• Amount of work hard to measure, so use alternative:– CYCLE TIME – look at cycle time for end

lane being the recycle bin

Rework due to late feedback by PM

• Will appear as high cycle times• Will appear as moving back cards on

the board (need to find way to measure)

• Can use special Class of service / card type to identify these kinds of stories better for measurement/tracking purposes

Workload compared to DEV

• See how much workload is in PM compared to DEV

• Look for trends and major changes that can indicate:– Bottleneck in PM– Idle and slack capacity – expect to see PM seeing

clients/customers at those times