slay the dragons of agile measurement

AW4Session6/8/1611:30AM

SlaytheDragonsofAgileMeasurement

Presentedby:

LarryMaccherone

AgileCraft

Broughttoyouby:

350CorporateWay,Suite400,OrangePark,FL32073888---268---8770··[email protected]://www.techwell.com/

LarryMaccheroneAgileCraftAnindustry-recognizedleaderinagile,metrics,andvisualization,LarryMaccheronecurrentlyhelpsanumberofcompanieswiththedesignoftheiranalyticsproductsincludingAgileCraftandPendo.io.Previously,LarryledtheinsightsproductlineatRallySoftwarewhichenabledbetterdecisionswithdata,leveragedbigdatatechniquestoconductgroundbreakingresearch,andofferedthefirst-everagileperformancebenchmarkingcapability.BeforeRally,LarryworkedattheSoftwareEngineeringInstituteforsevenyearsconductingresearchonsoftwareengineeringmetricswithaparticularfocusonreintroducingquantitativeinsightbackintotheagilecommunities.

https://LinkedIn.com/in/LarryMaccherone

The Eight

Dragons of

Agile

Measurement

Topic presented by:

Larry Maccherone LinkedIn.com/in/LarryMaccherone


Jean

Tabaka She taught us how

to collaborate. She

made us all better.

She made us want

to be better. The

agile community

has lost a most

insightful, caring,

inspiring person.


Larry Maccherone LinkedIn.com/in/LarryMaccherone


Using measurement in

an agile environment

can be dangerous


The Dragons of Agile Measurement If you do metrics wrong, you will harm your agile transformation

1. Dragon: Measurement as a lever

Slayer: Measurement as feedback

2. Dragon: Unbalanced metrics

Slayer: 1 each for Do it

fast/right/on-time, and Keep doing it

3. Dragon: Metrics can replace

thinking

Slayer: Metrics compliment thinking

4. Dragon: Expensive metrics

Slayer: 1st work with the data you

are already passively gathering

5. Dragon: Using a convenient

metric

Slayer: Outcomes Decisions

Insight Metric (ODIM)

6. Dragon: Bad analysis

Slayer: Simple stats and

simulation

7. Dragon: Single outcome forecasts

Slayer: Forecasts w/ probability

8. Dragon: Human emotion and bias

Slayer: Tricks to avoid your own

biases and overcome those of

others


Dragon #1

Using measurement as a

lever to directly drive

behavior


Manipulating

Others

Dragon #1

Using metrics as a

lever to drive

someone else’s

behavior


Self

Improvement

Dragon slayer #1

Using metrics to

reflect on your own

performance


Dragon #2

Unbalanced Metrics


Do it fast

Do it on

time

Do it right

Keep doing

it

Balanced scorecard of metrics

at least one from each quadrant


Dragon #3

Metrics can replace thinking


Dragon #4

Expensive metrics


Favor metrics derived from

existing records


Dragon #5

Using a convenient metric

aka “Lamp post metrics”


Good players?

Monta Ellis

9th highest scorer

(8th last season)

Carmelo Anthony (Melo)

8th highest scorer

(3rd last season)


Not everything

that counts

can be counted,

and

not everything

that can be

counted,

counts.


Dragon slayer #5

ODIM

O U T C O M E

D E C I S I O N

I N S I G H T

M E A S U R E

THINK

EFFECT

like Vic Basili’s

Goal-Question-Metric (GQM)

but without

ISO/IEC 15939 baggage


Dragon #6

Bad analysis and

visualization


Top 10 criteria for great visualization

1. Answers the question,

"Compared with what?”

(SO What?)

2. Shows causality, or is at least

informed by it.

(NOW WHAT?)

3. Tells a story with whatever it

takes.

4. Is credible.

5. Has business value or impact in

its social context.

6. Shows

differences

easily.

7. Allows you to see the forest

AND the trees.

8. Informs along multiple

dimensions.

9. Leaves in the numbers where

possible.

10. Leaves out glitter.

Credits:

• Edward Tufte

• Stephen Few

• Gestalt (School of Psychology)

11. Uses good visual grammar


Time check?

Skip the next 8 slides if it’s

now past half the allotted time.

They go over the elements of

good visual grammar.


11. Good visual grammar (1)

When to use a straight line series?

1. Cumulative data

(later data points include

items from earlier data points)

example: Burn or Scope

series

2. Benchmark for column series

3. Connected by “ordinal inertia”

(ordinal usually = temporal)

However, spline is usually

what you want in this

instance.



When to use a dashed line?

1. “Ideal” or reference

2. Forecast

3. Regression or fit

line



When to use a column (vertical bar)

series?

1. Counts/sums independent of

neighbors

example: Throughput or

velocity chart



When to use a bar (horizontal)

series?

(Rarely)

1. The x-axis variable is causal?

2. In a table

3. Stylistic reasons

• Length of labels

• Two-variable display



When to use stacked area?

1. Multiple cumulative series

(the ordinal sum means

something)

example: a cumulative

flow diagram (CFD)



When to use a single area series or

multiple (non-cumulative) area

series?

NEVER?



When to use a single spline series?

1. Filling in for missing granularity

2. Rolling average



When to use a single scatter series?

1. Two variables that are

imperfectly

correlated


Dragon #7

Single outcome forecasts


Monte Carlo Forecasting What it looks like

Live demo: http://lumenize.com (use Chrome)

http://lumenize.com/


Seek to

change the nature of

the conversation


Every decision is a

forecast!


You are forecasting that

your choice will have better

outcomes than the other

alternatives


So…

quality of decision depends upon:

1. alternatives considered, and

2. models used to forecast the

outcome of those alternatives.

Probabilistic models are superior


For a given alternative, let:

Pg = Probability of good thing happening

Vg = “Value” of good thing happening

Then:

Value of the alternative = Pg × Vg


An

lean/agile product

management example


$8M

Best case

(25%)

$1M

Likely case

(50%)

$1M

Worst case

(25%)

1

$2M $2M $1M

2

Which strategy is best…

…for your company?

PW × VW = .25 × -$1.00M = -$0.25M

PL × VL = .50 × $1.00M = $0.50M

PB × VB = .25 × $8.00M = $2.00M

-----------

$2.25M

…for your career?

PW × VW = .25 × $1.00M = $0.25M

PL × VL = .50 × $2.00M = $1.00M

PB × VB = .25 × $2.00M = $0.50M

-----------

$1.75M


If you get only 1 project then

strategy 2 is better

75% of the time

If you get ∞ projects then

strategy 1 is better

100% of the time

How many projects do you need for

strategy 1 to be better

more often than not?


Play with it yourself at:

http://jsfiddle.net/lmaccherone/j3wh61r7/


Dragon #8

Human emotion and bias

https://LinkedIn.com/in/LarryMaccherone @LMaccherone @TheAgileCraft

We don't see things

the way they are.

We see things

the way we are.

~The Talmud

Men's Airo Outdoor Shoes


Getting

probability

input you

can trust


Trained/Calibrated

Untrained/Uncalibrated

Statistical Error

“Ideal” Confidence

30%

40%

50%

60%

70%

80%

90%

100%

50% 60% 80% 90% 100%

25

75 71 65 58

21

17

68 152

65

45 21

70%

Assessed Chance Of Being Correct

Perc

ent C

orr

ect

99 # of Responses

We are inaccurate when assessing probabilities

Copyright HDR 2007

[email protected]

But, training can “calibrate” people so that of all the times they

say they are X% confident, they will be right X% of the time


Equivalent Bet calibration

What year did Newton published the Universal Laws of

Gravitation?

Pick year range that you are 90% certain it would fall within.

Win $1,000:

1. It is within your range; or

2. You spin this wheel and it lands green

Adjust your range until 1 and 2 seem equal.

Even pretending to bet money works.

90%

10%


Now what?

• Questions?

• Day-long seminar on agile metrics

• Workshop to design your own

metrics regimen

• AgileCraft Demo

• Make contact with me on LinkedIn

https://linkedin.com/in/larrymaccherone

Now what?

• Meet me at the AgileCraft booth to

answer questions






“They” say…

Nobody knows what’s gonna happen

next: not on a freeway, not in an

airplane, not inside our own bodies

and certainly not on a racetrack with

40 other infantile egomaniacs.

– Days of Thunder

Trying to predict the future is like

trying to drive down a country road

at night with no lights while looking

out the back window.

– Peter Drucker

Never make predictions, especially

about the future.

– Casey Stengel


When you come to a

fork in the road…

take it!

~Yogi Berra


Now what?

• Questions?

• Day-long seminar on agile metrics

• Workshop to design your own

metrics regimen

• AgileCraft Demo



Now what?

• Meet me at the AgileCraft booth to

answer questions

• Share webinar recording of this

content to co-workers who couldn’t

be here https://attendee.gotowebinar.com/register

/5432450926944227330



https://attendee.gotowebinar.com/register/5432450926944227330




