non-pathological software metrics
DESCRIPTION
As semi-scientific software professionals, we like the idea of measuring our work. In some cases, our bosses like the idea much more than we do. Yet, meaningful software development metrics are notoriously challenging to define, and many people have given up trying because metrics often incentivize pathological behaviors. Since you get what you measure, most metrics lead development teams to optimize numeric proxies for success rather than the goals these proxies were intended to represent. Mindful of the pitfalls of quantifying knowledge work, Stephen Frein examines some typical software development metrics and why measuring them tends to be harmful. Then he recommends alternate metrics—such as “truck number”—that give software development teams a chance to engage in meaningful measurement with a lower risk of harmful side effects. If you want or need to measure your team’s success without ruining it, join Stephen for helpful advice.TRANSCRIPT
BT4 Session 6/6/2013 10:15 AM
"Non-Pathological Software Metrics"
Presented by:
Stephen Frein Comcast Cable
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Stephen Frein Comcast
Stephen Frein is a software development manager at Comcast, where his team creates tools for process automation and quality assurance. As an adjunct professor at Drexel University, he delivers soporific lectures on database development and IT management. Stephen has been leading development teams for fifteen years, and occasionally does it well, mostly by dint of accidents he cannot reliably replicate. He has presented at previous conferences by sneaking into unused rooms and deceiving the unsuspecting. Stephen enjoys polluting the hive mind via frein.com and writing vapid, self-deprecating bios.
Non-PathologicalNon PathologicalSoftware Metrics
Stephen FreinC tComcast
http://[email protected]
About Me
Context MattersContext Matters
Metrics can do harm.
B f l f h Be careful of what you incentivize.
Why Measure?
ObjectivityClarity
Objectivity
Authority
What to Measure?
Productivity!
You can't do that.Quit it.
W h f bl
Stop it.-- Joel Spolsky
We have no way of reasonably measuring productivity.
-- Martin Fowler
You get what you measure.
I th t h t t?Is that what you want?
LOCLOCLines of Code
More is Better!
public Cost calculateShipmentCost(Order someOrder) {Cost shipCost = new Cost(0);if (someOrder.getTotalCost() < 25) { shipCost.setCost(10); }else { shipCost.setCost(5); }return shipCost;
} 9 LOC}
public Cost calculateTotalCost(Order someOrder) {return someOrder.getTotalCost().add(
calculateShipmentCost(someOrder).getCost() );}
public Cost calculateReturnRefund(Order someOrder) {// only half of shipping costs get refundedreturn someOrder.getTotalCost().add(
calculateShipmentCost(someOrder).getCost()/2) );}
public Cost calculateTotalCost(Order someOrder) {Cost shipCost = new Cost(0);if (someOrder.getTotalCost() < 25) { shipCost.setCost(10); }else { shipCost.setCost(5.00); }return someOrder.getTotalCost().add(
shipCost getCost() );shipCost.getCost() );}
public Cost calculateReturnRefund(Order someOrder) {Cost shipCost = new Cost(0);if (someOrder.getTotalCost() < 25) { shipCost.setCost(10); }else { shipCost.setCost(5.00); }{ p ( ) }// only half of shipping costs get refundedreturn someOrder.getTotalCost().add(
shipCost.getCost()/2 );} 10 LOC
Trade-offs
“Productivity”Maintainability ProductivityMaintainability
Change Incentives
EconomicsDrive Behavior
Don’t make a measureGoodhart’s Law
into a target.
Guess when …
This never would have happened if
i
<sarcasm>
we were using story points.</sarcasm>
Predicting the Future
“Ideal”
Metrics that focus on a proxy often
cause pathological behaviors.
PROXY GOAL
The proxy loses its connection to the intended goal.
What to do?
Balance
Test Cases Per Hour Defects Per Test Case
Stealth
Voyeurism
Independence
Metrics I’ve Tried
Truck NumberHow many ypeople can get hit by a truck before the project is in big project is in big trouble?
Customer SatisfactionWhat do your What do your customers think of your product?
Recommended:Sh rto Short
o Simpleo Anonymouso Comment box
Learning
Find the Incentives
Questions?