1

Understanding Metrics for Agile Teams

Bimlesh Gundurao

Sept 26 2013

2

A Business, Technology and Talent Development Consulting

Company with focus on Healthcare , Retail & IT

Business

Technology

People

Vision

To become the most

preferred business

partner to our

customers through

leadership in our

actions, values and

social responsibility

Mission

To be a world class

organization in enabling

clients to become

Leaders in their industry

Values

LEAD by Example

Leadership, Empower, Agile, Decisive

3

are you measuring your performances ?

4

Agile Manifesto & Scrum Framework

Time box

Inspect No Changes

Adapt

Commit

5

Lean Principles

• Eliminate waste

• Build quality in

• Create knowledge

• Defer commitment

• Deliver fast

• Respect people

• Optimize the whole

6

self-induced vs. enforced

internal vs. external

snapshot vs. forward looking

different perspectives

different perspectives

7

effort time scope quality

Classic Metrics

8

Why do Metrics matter?

They provide true reflection

9

some Agile principles:

Create Value for the customer as early as possible

Eliminate Waste (WIP, YAGNI)

Drive and Respond to Change, quickly

Time/Capacity Boxing (see Scrum and Kanban)

Provide Visibility into project progress

Enter Agile

10

Why do Metrics matter

REASONS #1 & #2

• We Have to!

• Self Defense!

Every stakeholder wants to know what’s going

on through a quantified measure!

11

Why do Metrics Matter?

Reason #3 To Make Business Decisions

• Decision making frequency increases multi-fold

• Such as

- Should we start this effort

- Which team needs the most help now

- When do we stop doing this product backlog

- Do we understand the customer better

- Did it actually help to remove that

- impediment

12

Why do Metrics Matter?

Reason #4

• To get feedback, so that forward-looking guesses have a higher

probability of being right

• We make a guess (aka estimate), and then we check later how

good the guess was

• If it is off a lot...maybe: ”gee, we need to learn how to estimate

better”

13

Why do Metrics Matter?

Reasons #5

• To change behavior...

– Not just the key business-decisions

– But as close as possible to all the behavior on a day-to-day

basis

14

What you can’t control you can’t manage!

What you can’t manage you can’t measure!

15

Defects

Code

Architecture

Usability

Documentation

Installation

Support

etc.

Quality

Metrics Explosion

16

Choosing the right Metrics

1. Goal Setting

2. Vital Few Vs Trivial

Many

17

too many KPIs are useless

18

Goal Setting

Answer 3 questions – What is its purpose?

– How will you report?

– How will you gather data?

19

Product Life Cycle

Metrics Measured at

different stages has

different value

20

Planning Onion

21

The product owner plans the product in

layers

© 2006-2007 Jeff Patton, All rights reserved,

www.agileproductdesign.com

Product

or Project

What business objectives

will the product fulfill?

Product Charter

Elevator Pitch

Release

How can we release

value incrementally?

What subset of business

objectives will each

release achieve?

What user constituencies

will the release serve?

What general capabilities

(big stories) will the

release offer?

Release plan

Iteration

What specifically will we

build? (user stories)

How will this iteration

move us toward release

objectives?

Iteration Plan

Story (Backlog Item)

What user or stakeholder need will

the story serve?

How will it specifically look and

behave?

How will I determine if it’s

completed?

Story Details

Acceptance Tests

22

The Planning Onion can grow to include product

portfolios and business strategy

© 2006-2007 Jeff Patton, All rights reserved,

www.agileproductdesign.com

Product

or Project

What business objectives

will the product fulfill?

Product Charter

Elevator Pitch

Release

How can we release

value incrementally?

What subset of business

objectives will each

release achieve?

What user constituencies

will the release serve?

What general capabilities

(big stories) will the

release offer?

Release plan

Iteration

What specifically will we

build? (user stories)

How will this iteration

move us toward release

objectives?

Iteration Plan

Story (Backlog Item)

What user or stakeholder need will

the story serve?

How will it specifically look and

behave?

How will I determine if it’s

completed?

Story Details

Acceptance Tests

Product or Project

Release

Iteration

Story

23

The Planning Onion can grow to include

product portfolios and business strategy

© 2006-2007 Jeff Patton, All rights reserved,

www.agileproductdesign.com

Product or Project

Release

Iteration

Story

24

The Planning Onion can grow to include

product portfolios and business strategy

Product or Project

Release

Iteration

Story

Product Portfolio

Business Strategy

© 2006-2007 Jeff Patton, All rights reserved, www.agileproductdesign.com

Daily by team

member

Bi-weekly by team

Quarterly by PO and Team

Bi Yearly by PO

Yearly by PO

25

The Planning Onion can grow to include

product portfolios and business strategy

Product or Project

Release

Iteration

Story

Product Portfolio

Business Strategy

© 2006-2007 Jeff Patton, All rights reserved, www.agileproductdesign.com

DOD

Velocity

Business Value

NPS

ROI

Profitability

26

Measuring Agile Team Maturity

Storming

Performing

Forming

Norming Disperse

Its all about

Conversations

and Behaviors

27

`

Peter Drucker

28

The Team wants metrics. Why?

• To help them see their work

• To plan with

• To determine when successful

• To push back on magical-thinking managers

• To challenge themselves

29

Some key attitudes

• We accept that things always were and always will be imperfect

• We relentlessly pursue perfection

30

31

32

The Agile approach

• Truth and transparency are essential

• The metrics are first for the Team

• Typically, we trust the Team

33

The Agile approach - 2

• Managers can visit a team at any time to see the meaning of

any numbers

• Managers have the patience and respect to observe the

Gemba

34

keep it simple, one step at a time

35

Good Metrics

Vital Few Measure

Results not Output

Measure Trends

Easy to Collect

Amplify Learning

Reinforce Desired

Behavior

1. Be accurate enough to enable better

decision making

2. Enable better actions and serious

improvement

3. Not be seriously gamed (inaccurate);

ideally - “gaming” is actually better

behavior

4. Change the behavior of all members of

team and related managers

5. Motivate the team (or at least not de-

motivate)

6. Simple enough that they are done, and

used well

7. Enable optimizing the whole

36

Metrics & Myths

Metrics get stale over time, Objective is lost!

37

Metrics Perspectives from the Trenches

“I once worked for a team that started doing TDD, and we decided

to measure the number of unit tests written during a sprint. The

metric reminded us of the fact that we wanted to write tests, and it

provided opportunities for celebrating as a team that we were

implementing our decision. We weren't judged by outside the team

on the metric, though, we didn't try to maximize it, and as soon as

writing tests became an integral part of our work, we abandoned

the metric, because it wasn't useful anymore”

38

Metrics should not scare or threaten people

Enforced metrics are often cheated or ignored

39

Agile Metrics

• Velocity

• Stories Completed (done, done)

• Number of Passing Unit and

Functional Tests (today or with

growth trend)

• Bugs open today

• % BV completed (if use BV

points or similar)

• Full Product Backlog (remaining

stories)

• Impediments Open Vs Resolved

• % Change in Velocity since

(inception, last year)

• Number of story points

completed to date; % of total.

• Bugs that escaped the Sprint

• Oldest bug open (with Sev

level)

• Sprints with stories incomplete

• Sprints with added stories

• Unplanned tasks (in the X

Sprint); related hours

40

Agile Metrics – More

• Stories added to / subtracted from the

Release

• Age of each story to done, done;

average age

• Impediments removed to date

• Builds that passed/failed initially, to

date

• Defects identified after done, done

• Defects identified after release

• If start with big bug list

– Bugs added (old features) (per time)

– Old Bugs resolved / closed (per time)

– Old Bugs remaining (over time)

• If starting with minimal automated

tests

– Number of automated tests (unit, functional, etc)

– Number of manual tests (that could be automated)

– Effort on manual testing

• Metrics around quality of builds and

regression tests

• Metrics around quality of code (eg,

cyclomatic complexity)

• Code coverage by automated tests

(unit, functional, etc.)

41

Cycle Time = Number of Things in Process/ Average Completion Rate

Little’s Law

time spent in each lane ?

bottlenecks ?

cycle time

lead time

Flow = Speed * Density, Density Speed => Traffic Jam

40 60 25

ouch!

ouch!

Some Kanban Specific Metrics

42

What metrics are these organizations using?

43

What Metrics works for you is important!

Case Study 1

• Rework Ratio

• Defects closed v/s resource capacity

• New defects injected (open + closed – monthly iterations)

• Defect per dev per week (Injected)

• Review Yield = (Defects captured in Review)/(Total defects

captured in review + testing)

• RCA (for client defects and return\rework defects from iterations)

and corrective actions

• Avg. Story points (expected v/s actual)

44

What Metrics works for you is important!

Case Study 2

• At completion of sprint, sprint status(g/y/r) and overall project

health(g/y/r)

– The above measure is dependent on the # of US planned versus done; if US not ‘done’, its not accounted in the completed US

– Project health depends not only on US done, but on cross functional dependencies(internal + external) and risks status. If any of those aren’t Green, project health is Y or R.

• Bug status reported at sprint completion

• Stories Progress - Hours completed vs remaining

• Stories Progress - how much work remains on each story, whats

the progress towards completing the work on each story

45

What Metrics works for you is important!

Case Study 3

• Project:

– Burn-downs for Sprints

– Burn-ups for Releases (it provides greater visibility into the value that can be derived at a certain point in time)

– Defect Arrival and Kill Rates

– Team velocity (with an eye on variance over time. Reducing variance are signs of a stabilizing system)

• Program/Portfolio:

– Cycle times

– Lead times

– Throughput

– Wait times and more..

46

What Metrics works for you is important!

Case Study 4

Sprint Level

• Velocity

• Wastage in hours/Sprint

• Service Test Automation Pass rate (Program)

• Regression Pass Rate (Team wise & Program)

• User Story - DOD Completeness

• Code Coverage

Program Level

• Business Value Delivered vs Business Value Backlog

47

What Metrics works for you is important!

Case Study 5 • Release/ Sprint Level:

– % Progress.

– Acceptance Criteria Data.

– User Acceptance Test : Accepted Data.

– Rework data After UAT.

– Test cases Executed.

– Test Cases returned.

– Velocity of sprint.

– Actual Acceptance Planned Vs final accepted.

– The above which in turn will get the quality of the sprints.

• Program Plan levels:

– Number of Stories Mapped to each release/Sprint.

– Number of Stories for total release.

– Probable cost per sprint based on a Projected Acceptance criteria.

• Line Management perspective.

– No: of Sprints Planned.

– Total no: of Team members in reach release. FTE’s & Consultants.

– Total no: of Cross functional Team Members.

– Future Forecast of Team members.

– Ramp up data across sprints.

• From Customer Validation:

– CSI.

– % of Progress in each release.

– Cost per sprint / Cost per Release.

– Cost Burn down.

– Value Burn up

• Estimations

– Either use Effort in Time for each Story.

– In depth Sub task level effort.

• Team Level:

– Happiness Factor

• Large Scale Integration Projects:

– Integration Hand-offs

48

Measurement Dimensions

Value

(To Customer)

Predictability

(Schedule)

Collaboration

(Process)

Quality

(Product)

49

Basic Metrics

Value Predictability

Collaboration Quality

Customer

Surveys Velocity

Burn Up/

Burn Down

Story Cycle

Time

Technical

Debt

50

Extended Metrics

Value Predictability

Collaboration Quality

Customer

Surveys Velocity

Burn Up/

Burn Down

Story Cycle

Time

Work-in-

Progress

Technical

Debt

RTF/

Automated

Tests

Team

Surveys

Cost per

Sprint/

Point Real Value

Delivered

NPV/ ROI

NPS

Defects

Continuous

Improvement

51

TO SUMMARIZE

52

build a simple but effective dashboard

53

measure, evaluate, improve

54

communicate clearly

@you: are you getting this ?

55

communicate visually

56

use simple tools (but use them!)

57

transparency: Metrics visible to everyone

58

aim higher

59

Q & A

60

Thank You

Bimlesh Gundurao

+91-988 024 4406

bimlesh@aguaisolutions.com

www.aguaisolutions.com

Twitter - @bimleshgundurao