mike ennis - managing the end game of a software project
TRANSCRIPT
Copyright © 2007 Accenture All Rights Reserved. Accenture, its logo, and High Performance Delivered are trademarks of Accenture.
*** CONFIDENTIAL ***
This presentation contains Accenture proprietary information that is being considered for patent protection. External use of this presentation may jeopardize Accenture intellectual property rights relative to the assets described herein. Please obtain approval from Accenture Legal & Commercial before sharing this material
externally.
Measuring & Managing the End Game of a Software Project
Mike Ennis, Senior Test ManagerEuroSTAR Conference 2009
2
Outline
The Problem
Is your Release Criteria meeting your needs?
Deciding what to measure (The Big Six)
How the metrics relate to one another
Setting ranges for each measurement
Creating your readiness “spider” chart
Managing the risks
Are you ready to release?
Benefits & Gotchas
Tools of the trade
Keys to Success
3
The Problem
How do you know when a product is ready to ship?
Release Criteria
Quality Metrics
Customer Commitment
Release dates are preset
Indefinite Testing
Adequate Test Coverage
Time & Resources
4
Is your Release Criteria meeting your needs?
No open critical or high defects
Minimal number of medium & low defects that have been approved by Senior Management
Product is able to run for 72 consecutive hours
No open installation or configuration issues
All pre-defined performance criteria has been met
5
Deciding what to measure (The Big Six)
Test Completion Rate
• The percentage of tests that have been attempted during the test cycle
Test Success Rate
• The percentage of tests that have passed during the test cycle
Total Open Defects
• The number of open defects that are currently logged against the product
Defects Found this week
• The number of defects found in a given day, week or build
• Also referred to as Defect Arrival Rate
Code Turmoil
• The percentage of code that has changed in a given day, week or build
• Also referred to as Code Churn
Code Coverage
• The percentage of line and path coverage that your test conditions satisfy
6
How the metrics relate to one another
Test Completion Rate
Test Success Rate
Code Coverage Rate
Total Open Defects
Defect Arrival Rate
Code Turmoil
7
Setting ranges for each metric (6 to 1)
What’s an acceptable Test Completion Percentage?
• 6 = 100% - 95%, 5 = 94% - 90%, …1 = < 74%
What’s an acceptable Test Success Percentage?
• 6 = 100%, 5 = 99% - 97%, …1 = < 87%
What’s an acceptable number of Total Open Defects?
• 6 = 0 – 10, 5 = 11 – 15, …1 = > 31
What’s an acceptable number of Defects Found this week?
• 6 = 0 -2, 5 = 3 -4, …1 => 11
What’s an acceptable percentage of Code Turmoil Percentage?
• 6 = 0 – 10%, 5 = 11 – 15%, …1 = > 30%
What’s an acceptable percentage of Code Coverage Percentage?
• 6 = 100 – 80%, 5 = 79 – 70%, …1 = < 39%
8
Creating & analyzing your readiness “spider” chart
RATING Ranges
Code Turmoil 36 = 0 - 10%, 5 = 11 - 15%, 4 = 16 - 20%, 3 = 21 - 25%,25 = 26 - 30%, 1 > 30%
Defects found this week 3 6 = 0 - 2, 5 = 3 - 4, 4 = 5 - 6, 3 = 7 - 8, 2 = 9 - 10, 1 > 11
Total Open Defects 2 6 = 0 - 10, 5 = 11 - 15, 4 = 16 - 20, 3 = 21 - 25, 2 = 26 - 30, 1 = > 31
Code Coverage 66 = 100 - 80%, 5 = 79 - 70%, 4 = 69 - 60%, 3 = 59 - 50%, 2 = 49 - 40%, 1 = < 39%
Test Success % 46 = 100%, 5 = 99% - 97%, 4 = 96% - 94%, 3 = 93% - 91%, 2 = 90% - 88%, 1 = < 87%
Test Completion % 66 = 100% - 95%, 5 = 94% - 90%, 4 = 89% - 85%, 3 = 84% - 80%, 2 = 79% - 75%, 1 = < 74%
TOTAL RATING 24
GREEN = 36 - 30, YELLOW = 29 - 18,
RED < 17
Raw Data 29-Apr-05 6-May-05 13-May-05 20-May-05 25-May-05 1-Jun-05 8-Jun-05
Code Turmoil % 25% 30% 15% 20% 18% 17% 23%
Defects found this week 5 13 3 3 3 2 7
Total Open Defects 5 18 10 15 18 20 27
Code Coverage % 0% 0% 80% 80% 80% 80% 80%
Test Success % 95.18% 93.38% 94.37% 94.37% 94.37% 94.37% 94.37%
Test Completion % 88.30% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
9
Creating & analyzing your readiness “spider” chart
10www.spinstitute.org 10© IISP, 1996-2008
•Analyzing your readiness “spider” charts over time
Product vx.0 7 weeks to release
0123456
Code Turmoil
Defects found this week*
Total Open Defects
Code Coverage Rate
Test Success Rate
Test Execution Rate
•Product vx.0 3 weeks to release
•0•1•2•3•4•5•6
•Code Turmoil
•Defects found this week*
•Total Open Defects
•Code Coverage Rate
•Test Success Rate
•Test Execution Rate
•Product vx.0 •1 week to release
•4
•5
•6•Code Turmoil
•Defects found this week*
•Total Open Defects
•Code Coverage Rate
•Test Success Rate
•Test Execution Rate
11
Creating & analyzing your readiness “spider” chartAnalyzing your readiness “spider” charts over time
Product vx.0 7 weeks to release
0123456
Code Turmoil
Defects found this week*
Total Open Defects
Code Coverage Rate
Test Success Rate
Test Execution Rate
•Product vx.0 3 weeks to release
•0•1•2•3•4•5•6
•Code Turmoil
•Defects found this week*
•Total Open Defects
•Code Coverage Rate
•Test Success Rate
•Test Execution Rate
•Product vx.0 •1 week to release
•4
•5
•6•Code Turmoil
•Defects found this week*
•Total Open Defects
•Code Coverage Rate
•Test Success Rate
•Test Execution Rate
12
Managing the risks
Understand the relationships between the metrics
Learn to anticipate and minimize the risks before they happen
Always know the information behind the data
13
Are you ready to release?
Redefine your Release Criteria using the individual/overall rating scale
Use colors for presentation & effectiveness
Let the data speak for itself
14
Benefits
Benefits
• Combines critical success factors in an easy to read/interpret spider chart
• Gives a good indication of product readiness as you get closer to your release date
• Identifies areas of risk quickly and shows their impact on overall quality
• Gives management the necessary data to make risk assessments
15
Gotchas
Gotchas
Gain buy in from key stake holders
Without information behind the numbers, the data can at times be misleading
Test case execution must be documented and repeatable
Test status reporting must be consistent
Configuration Management must be consistent
Source code repository must be reliable
Allow for exceptions
Be consistent
16
Tools of the trade
Test Management & Defect Tracking
• HP/Mercury Quality Center or TestDirector
• IBM Test Manager
Source Control
• IBM Rational Clearcase
• CVS, PCVS
• McCabe IQ
Code Coverage
• McCabe IQ
• IBM Rational PureCoverage
17
Keys to Success
Keep it simple
Communicate early & often
Stay flexible
Solve issues as a team
Manage the risks before they manage you
