creating meaningful defect metrics by harmony brenner
DESCRIPTION
This is a presentation on how to create and implement meaningful defect metrics for software projects.TRANSCRIPT
Creating Meaningful Defect Metrics
By Harmony BrennerJanuary 21, 2014
“Measurement is the first step that leads to control and eventually to improvement.
If you can’t measure something, you can’t understand it.
If you can’t understand it, you can’t control it.
If you can’t control it, you can’t improve it.”
― H. James Harrington
AgendaGoalsDefinitionsPre-RequisitesProcess RequirementsMetrics and ExclusionsCalculate DRE MetricsCalculate Rework MetricsPresenting ResultsMetrics Usage and Sharing
GoalsIncrease attention given to all phases of
testing (Unit, System, User Acceptance) to improve quality.
Decrease number of defects detected in later phases of testing to reduce rework and overall costs.
Implement metrics as an objective method to measure outcomes and improvements over time.
DefinitionsDRE – Defect Removal Effectiveness.
Metrics – Use of numerical ratings to determine quality and performance of item / phase being measured.
Also define any Testing phases and describe who / which team(s) own accountability.Examples – SIT is usually done by IT Team;
UAT is usually performed by Business Team
Pre-Requisites
Ensure your Defect Tracking system contains discrete categories for the following:
StatusesReasons / outcomes for defect closureImpact LevelsRoot CausesAbility to track Cycles of Rework
Process RequirementsEncourage consistent creation and updates of
defects by IT and Business teams immediately upon discovery.Mentor teams how to create effective defect
reports.
Foster communication between teams to ensure that all release defects are worked through to completion.Defect Tracking TrainingResolve other barriers to acceptance
Metrics and Exclusions
Decide and describe whether any Defect Statuses should be excluded from measurement.
Example – statuses indicate the defect should not have been created may merit exclusion.
Cancelled, No Action Required, etc.
Metrics and Exclusions (cont’d)
Decide and describe whether any Defect Root Causes should be excluded from measurement.
Example – categories that indicate the defect should not have been created may merit exclusion.
Change Request, User Error, etc.
Calculate DRE MetricsCalculate the DRE (Defect Removal Effectiveness)
for each test phase in the project to be measured:
Phase DRE = Phase Defects / (Phase Defects + All Defects)
SIT Defects (10) / (SIT Defects (10) + UAT Defects (5) )
10 / 15 = 67% is SIT DRE 5 / 15 = 33% is UAT DRE
Calculate Rework MetricsDecide how to track Rework instances for
defects.
Example – Audit trail may count changes between Retest vs. Rework statuses.
Where possible, automate Rework counts per defect so these can be calculated vs. manually determined.
Presenting ResultsShare results in a non-judgmental
framework.
Anonymize and de-personalize outcomes wherever possible. This is team, not individual, performance.
Give credit where due and be a change champion!
Metrics Usage and SharingAlways tell yourself the truth!
Use metrics to raise awareness on overall quality – not as punishment!
Share metrics at regular intervals – no surprises.
Compare metrics over time on the same project or among different projects to measure performance.
Want to Learn More?
I enjoy professional discourse on a variety of subjects pertaining to Software Quality and Testing.
Harmony Brenner – [email protected]
LinkedIn – www.linkedin.com/in/harmonybrenner