4/16/2013

1

1

Lee CopelandSoftware Quality Engineering

lee@sqe.com

The Mismeasure of Software

The Last Talk on Measurement You’ll Ever Need to Hear

22

Topics

• What is measurement?

• The “Four Don’ts” of measurement

• My favorite indicator—isn’t a measurement

• Lee’s Zeroth Law of Measurement

4/16/2013

2

33

What Is Measurement?

• “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them”

–Cem Kaner

44

A Classic Example

• Phrenology

– The brain is the organ of the mind

– The brain is not a homogenous entity, but an aggregate of “mental organs” with specific functions

– These organs are topographically localized

4/16/2013

3

55

A Classic Example

• Phrenology

– The relative size of any particular mental organ indicates the power or strength of that organ

– External measurement can be used to determine the internal states of our mental powers

66

A Volunteer From The Audience?

4/16/2013

4

77

A Classic Example

• Even automated devices for phrenological analysis were created. The psychographnetted its inventors about $200,000 at the 1934 Century of Progress Exposition in Chicago

• Some employers even required phrenological certificationsbefore hiring employees

88

What Is Measurement?

• Did you notice that Kaner’s definition only requires intent. It says nothing about the measurement actually being useful?

4/16/2013

5

99

Two Causes of Measurement Failure

• Map-territory error

• Reification error

1010

What Is Measurement?

• “Measurement is the empirical, objective assignment of numbers, according to a rule

derived from a model or theory, to attributes

of objects or events with the intent of describing them”

–Cem Kaner

4/16/2013

6

1111

Map-Territory Error

• A common error is the confusion of a model with reality

• “The map is not the territory”

– Alfred Korzybski, 1931

• Mathematical or simulation models may helpus understand a system or situation but real life always differs from the model

1212

Map-Territory Error

• The model / theory / map may just be WRONG!

4/16/2013

7

1313

What Is Measurement?

• “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of

objects or events with the intent of

describing them”

–Cem Kaner

1414

Reification Error

• A Reification error occurs when an abstraction (abstract belief or hypothetical construct) is treated as if it were a physical object or a real event

• Counting them is dangerous! Comparing the counts is even more dangerous!!!

4/16/2013

8

1515

Reification Error

• Examples include:

– Intelligence

– Performance

– Commitment

– Motivation

– Leadership

– Attitude

1616

Reification Error

• Now do you understand why you feel so violated after your Annual Performance Review?

4/16/2013

9

1717

But Wait, There’s More !

• Consider these reification errors:

– Requirements

– Test cases

– Defects

1818

Topics

• What is measurement?

• The “Four Don’ts” of measurement

• My favorite indicator—isn’t a measurement

• Lee’s Zeroth Law of Measurement

4/16/2013

10

1919

The Four Don’ts

Don’t #1

Don’t #3

Don’t #2

Don’t #4

2020

Don’t #1

• Don’t measure it if you don’t know what it means

4/16/2013

11

2121

Don’t #1

• Does the number actually represent the attribute we are trying to measure?

• Examples:

• Lines of code – program size

• Number of test cases – thoroughness

• Number of defects found – product quality

• Superclass:Subclass ratio – good systemdesign

2222

Don’t #2

Thanks for all your

hard work producing

this measurement

report

• Don’t measure it if you’re not going to do something because of the measurement

4/16/2013

12

2323

Don’t #2

• What does this tell your staff?

– You’re a clown

– You’re an idiot

– You’re an H

2424

Don’t #3

• Don’t turn your measurement into a goal

• Goodhart’s Law — Proposed in 1975 by Charles Goodhart—a former advisor to the Bank of England and Emeritus Professor at the London School of Economics

4/16/2013

13

2525

Don’t #3

• The law states that “once an indicator is made a target for the purpose of guiding policy, then it will lose the information content that originally made it useful”

• Goodhart drones on, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”

2626

Don’t #3

• Professor Marilyn Strathern has restated Goodhart's Law more succinctly and more generally:

“When a measure becomes a target, it ceases to be a good measure”

4/16/2013

14

2727

Don’t #3

• A classic metric of this type is “lines of code written per day.” When used as a measure, this metric can be valuable for estimating, planning, and tracking

• However, when LOC/day becomes a goal, developers may be enticed to write more lines of (less efficient) code

2828

Don’t #3

For example, a classic Java “for” loop has the generic form:

for (initialization;

termination; increment) {

statement(s)

}

An example of this loop is:

for (int i=1; i<11; i++) {System.out.println("Count is: " + i);}

4/16/2013

15

2929

Don’t #3

If writing more lines of code is rewarded, I can write the equivalent code as:

int i;i=1; while (i<11)

{System.out.println("Count is: " + i);i++;}

thus increasing the LOC count while providing the identical functionality

3030

Don’t #3

• Goodhart’s Law reminds us that when rewards and punishments are connected to the achievement of specific goals, unintended consequences are often the result

4/16/2013

16

3131

Don’t #3

• If LOC is too “old-school”, consider Velocity, a premier agile metric

• What have you seen happening with Velocity?

Goal

3232

Don’t #4

• Don’t focus on measuring effort– Number of test cases created– Number of test cases executed– Number of test cases passed– H

4/16/2013

17

3333

Don’t #4

• Instead, focus on measuring accomplishments– Number of defects discovered before

installation– Costs saved through testing– H

3434

Topics

• What is measurement?

• The “Four Don’ts” of measurement

• My favorite indicator—isn’t a measurement

• Lee’s Zeroth Law of Measurement

4/16/2013

18

3535

My Favorite Indicator

• Lines of code

• Percent test cases passed

• Defect density

• Defect detection percentage

3636

My Favorite Indicator

• My favorite indicator is a

4/16/2013

19

3737

My Favorite Indicator

is

3838

My Favorite Indicator Is

• Surprise

– “To discover suddenly and unexpectedly, without warning”

– “To become aware of something not previously perceived”

– “A spontaneous awareness of a discontinuity in our understanding”

4/16/2013

20

3939

My Favorite Indicator Is

• Surprise is an interesting indicator for use in software development

• We could examine the surprise itself

• We could examine the reason(s) for our surprise

4040

Satir Interaction Model

Virginia Satir

4/16/2013

21

4141

Intake

• Intake is the process of observation, of bringing part of the “world out there” into our consciousness

4242

Intake Surprises

• As humans, we fail to observe huge amounts of information; there is simply too much

• Some individuals and organizations maintain an “oblivious culture” – they chose not to systematically observe anything

4/16/2013

22

4343

Intake Surprises

• A second type observes, but quickly filters out data that does not match their world-view

• A third, to prevent having to deal with realities, actually prohibits observing

4444

Meaning

• Meaning is the process of mapping our observations to our context, experience, and feelings

• Biases, agendas, pressures, expectations, and even past unresolved personal issuescan cloud our assignment of meaning

4/16/2013

23

4545

Meaning Surprises

• If not careful, we may assume that the first meaning we assign is the correct meaning is the only meaning

• Remember Weinberg’s Rule of Three – “if you can’t think of at least three different meanings, you haven’t thought enough about it”

4646

Significance

• Significance is the process of deciding whether the observation and meaning are important enough to affect our actions

4/16/2013

24

4747

Significance Surprises

• We may fail to assign the proper significance because:

– the observation does not fit into our previous experience

– we just can’t evaluate its importance

– we may be operating under rules that do not serve us well

– we may have been conditioned to minimize the significance

4848

Significance Surprises

4/16/2013

25

4949

Surprise

• Focus on your processes of Intake, Meaning, and Significance

• To learn something, examine the surprise

• To learn even more, examine the reasons why you were surprised

5050

Topics

• What is measurement?

• The “Four Don’ts” of measurement

• My favorite indicator—isn’t a measurement

• Lee’s Zeroth Law of Measurement

4/16/2013

26

5151

Lee’s Zeroth Law of Measurement

5252

Lee’s Zeroth Law of Measurement

• “If you measure the wrong thing, and reward the wrong thing, don’t be surprised if you get the wrong thing”

FamousGuru

&Grandpa

4/16/2013

27

5353

Recap

• What is measurement?

• The “Four Don’ts” of measurement

• My favorite indicator—isn’t a measurement

• Lee’s Zeroth Law of Measurement

5454

Thanks

Shamelesscommercial

message

lee@sqe.com