lessons learned in project metrics · lessons learned in project metrics: are your metrics dumb or...
TRANSCRIPT
Leading Answers Inc. - Lessons Learned in Metrics
1Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
February 16th, 2010
Software QA Discussion Group
Mike Griffiths, PMP
Leading Answers Inc.
Lessons Learned in Project Metrics:Are Your Metrics Dumb or Smart?
www.LeadingAnswers.com
My Background
• Project Manager and Trainer• >20 years IT experience on utilities, defense, & finance
• 8 years PMO Agile-to-Traditional Integration
• Agile Project Management• Helped create Agile method DSDM in 1994
• 16 years agile project experience (DSDM, XP, Scrum, FDD)
• Board director of Agile Alliance and APLN
• Author, trainer, and presenter Agile Conference 2001-9
• Traditional Project Management• PMP, PRINCE2 certifications
• PMBOK v3 contributor and reviewer
• Trainer for PMI SeminarsWorld 2005-2010
• Presenter PMI Global Congress 2004-2009
Leading Answers Inc. - Lessons Learned in Metrics
2Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
• 1 <Star>
• 2
• 3
• 4
• 5
Agenda
4) Measuring Up
3) “Design Factory” Metrics
2) The Hawthorne Effect
1) Measurement Troubles
6) Smart Metric examples
5) Leading vs. Lagging Metrics
Metrics are like fire…
Leading Answers Inc. - Lessons Learned in Metrics
3Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Measurement
Not all observations are useful: • The sun rises up in the sky in the morning and then falls down again at night
• Planets revolve around the earth
• Stars come out at night
• Heavier objects fall faster than lighter objects
”There are so many possible measures in a software process that some random selection of metrics will not likely turn up something of value” – Watts Humphrey
Not all useful things are easily observable: • Spouse’s mood
• Sponsor Confidence
• Team Commitment
"Not everything that can be counted counts, and not everything that counts can be counted“ – Albert Einstein.
The Hawthorne Effect
You will influence what you measure...
Leading Answers Inc. - Lessons Learned in Metrics
4Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Metrics Should be:
1. Simple, self-generating
2. Relevant to the end-goal
3. Ideally, leading, future-focused
Design Factory Metrics
Traditional Metrics Examined
Desirable Characteristics: The Hawthorne Effect is positive
Simple, self generating
Relevant to the end-goal
Leading, future-focused
Traditional Metrics:• Lines of Code Written – poor, does not reward
simplification, leads to code bloat
• Function Points Delivered – poor, effort to generate, not relevant to the end-goal of project
• Hours Worked – poor, leads to long hours, burn-out, defects, consumed budgets
Leading Answers Inc. - Lessons Learned in Metrics
5Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Smarter Project Metrics
Since you will influence what you measure…
Choose Smarter Metrics:
• Features Accepted
• Sponsor Confidence
• User Satisfaction
• Defect Cycle Times
Smarter Metrics
• Features Accepted
Not StartedIn ProgressAccepted
Scope (points) IPS Project Progress Spend
0
500
1000
1500
2000
2500
3000
1/10
/06
1/12
/06
1/2/
07
1/4/
07
1/6/
07
1/8/
07
1/10
/07
1/12
/07
1/2/
08
1/4/
08
1/6/
08
1/8/
08
1/10
/08
1/12
/08
1/2/
09
1/4/
09
1/6/
09
1/8/
09
1/10
/09
1/12
/09
Scope BuiltAccepted
Leading Answers Inc. - Lessons Learned in Metrics
6Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Smarter Metrics
• Features Accepted
0
50
100
150
200
250
300
350
400
450
Jan Feb Mar Apr May Jun Jul Aug
Time
Featu
res
Not Started
In Progress
Completed
Not StartedIn ProgressAccepted
Feature Based Reporting
Example 2: Parking lot Diagrams
Feature Set Status: Enter Order
Details
(15)
24%Percentage Complete:
Not Started
Work in Progress
Completed
Attention (i.e. behind)
Percentage Complete Bar
Mar 2010Target Completion Month:
Not Complete Complete
FB Chief Programmers Initials
Feature Set Name
Number of Features in
the Feature Set
The Feature Set called Enter Order
Details has 15 features and is
currently 24% complete. It is due to
be completed by March 2010
Leading Answers Inc. - Lessons Learned in Metrics
7Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Create New
Order
(5)
100%
Capture
Customer
Details
(9)
75%
Enter Order
Details
(15)
24%
Process
Payment
(11)
Create New
Customer
(8)
55%
Amend
Customer
Details
(6)
20%
Archive
Customer
(4)
Stock
Search
(6)
95%
Item
Details
(12)
75%
Order Processing
Customer Management
Inventory Management
Jan 2010
Feb 2010
Mar 2010
Apr 2010
Jan 2010
Feb 2010
Feb 2010Feb 2010
Mar 2010
CM DH LF RS NC
AW
KB
SW SW
Feature Based Reporting
Smarter Metrics
User Satisfaction
Leading Answers Inc. - Lessons Learned in Metrics
8Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Smarter Metrics
• Sponsor Confidence
Smarter Metrics
• Defect Cycle Time
Leading Answers Inc. - Lessons Learned in Metrics
9Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
• “You get what you measure”
• “You get only what you measure, nothing else”
• “You tend to loose the things that you can’t measure: insight, collaboration, creativity”
• Nucor Steel - From startup in 1970 to North America’s largest and most profitable steel company
• Incentive pay based on productivity
• Measure one level above where expected / directly controllable
Measuring Up
• Defects could be traced back to individual developers, but may be environmental
• Rolling up to an entire team and get testers involved earlier to provide more timely and valuable feedback to developers
“Instead of making sure that people are measured within their span of control. It is more effective to measure people one level above their span of control. This is the best way to encourage teamwork, collaboration, and global, rather than local optimization” – Mary Poppendieck
Measuring Up in Software
Leading Answers Inc. - Lessons Learned in Metrics
10Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
• For an accountant, a perfect view of the past might be useful
• For a project manager, a perfect or even an imperfect view of the future is far more useful
Leading vs. Lagging Metrics
Pay less attention to Lagging Metrics
• Actual values
Pay more attention to Leading Metrics
• Trends
• Impacts of projections
Trends
CR = Change
Requests
Clarif. =
Clarifications
Observations Mar 3 Mar 10 Mar 17 Mar 24 Mar 31
Defects Opened 5 25 30 20 10
Defects Closed 1 16 35 22 15
Defects Remaining 4 13 8 6 1
CR Opened 0 18 20 23 12
CR Closed 0 11 21 16 9
CR Remaining 0 7 6 13 16
Clarif. Opened 9 12 14 8 2
Clarif. Closed 6 14 11 9 5
Clarif. Remaining 3 1 4 3 0
Total Observations 7 21 18 22 17
Observation Trends
0
2
4
6
8
10
12
14
16
18
Mar 3 Mar 10 Mar 17 Mar 24 Mar 31
Defects Remaining
CR Remaining
Clarif. Remaining
Leading Answers Inc. - Lessons Learned in Metrics
11Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Trends - Risks ReducedJan Feb Mar Apr
ID Risk Short Name Imp Prob Sev Imp Prob Sev Imp Prob Sev Imp Prob Sev
1 JDBC driver performance 3 2 6 3 0 0 3 0 0 3 0 0
2 Calling Oracle Stored Procs via web service 2 2 4 2 0 0 2 0 0 2 0 0
3 Remote app distribution to PDA's 3 2 6 3 1 3 3 0 0 3 0 0
4 Oracle Warehouse Builder stability 2 2 4 2 3 6 2 2 4 2 0 0
5 Source system availability 2 1 2 2 1 2 2 0 0 2 0 0
6 Access to user community 2 1 2 2 2 4 2 2 4 2 1 2
7 Availability of Architect 2 2 4 2 3 6 2 2 4 2 0 0
8 Server Upgrade necessary 1 2 2 1 1 1 1 0 0 1 0 0
9 Oracle Handheld Warehouse Browser Launch 3 1 3 3 1 3 3 3 9 3 1 3
10 PST changes for BC 0 0 0 0 0 0 2 2 4 2 1 2
33 25 25 7
Project Risk Profile
0
5
10
15
20
25
30
35
Jan Feb Mar Apr
PST changes for BC
Oracle Handheld Warehouse Browser Launch
Server Upgrade necessary
Availability of Architect
Access to user community
Source system availability
Oracle Warehouse Builder stability
Remote app distribution to PDA's
Calling Oracle Stored Procs via web service
JDBC driver performance
Analyst UI Designer Code & UT Sys. Test
45 30 40 35
• Identifying constraints
• Buffering
• Managing flow
Cycle Times
Leading Answers Inc. - Lessons Learned in Metrics
12Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Cycle Times
Construction Cumulative Flow
0
20
40
60
80
100
120
140
20/02/
2004
27/02/
2004
05/03/
2004
12/03/
2004
19/03/
2004
26/03/
2004
02/04/
2004
09/04/
2004
16/04/
2004
23/04/
2004
30/04/
2004
07/05/
2004
14/05/
2004
21/05/
2004
28/05/
2004
Total Features
Analysis
UI Design
Code & UT
Sys Test
}Widening
Area
Bottleneck Activity = “UI Design” (activity below widening area)
Little’s Law
Little’s Law:
Cycle times are proportional to queue lengths.
(We can predict completion times based on queue size)
Leading Answers Inc. - Lessons Learned in Metrics
13Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com
Task Boards & Kanban
Summary
1. Use metrics wisely
2. Leverage the Hawthorne Effect
3. Measure Up
4. Favour Leading Metrics
More info and slides: www.LeadingAnswers.com
Email: [email protected]