PSP Quality Strategy
[ SE-280Dr. Mark L. Hornick
1
[ SE-280Dr. Mark L. Hornick
2
What is software quality?
Basic definition: Meeting users' needs
What about “wants”? True needs may be hard to
establish
Hierarchy of needs Do required tasks Be
Usable & convenient Install quickly & easily Run consistently
Economical Timely Dependable
No destructive or unexpected behavior
[ SE-280Dr. Mark L. Hornick
3
How do you measure quality?
[ SE-280Dr. Mark L. Hornick
4
In the PSP, defects are the basic quality measure; is this reasonable?
On the one hand, defects are only important to the customer or user if they cause visible adverse effects.
• Affect operations• Cause inconvenience• Cost time or money• Result in a loss of confidence
However, defects have a big impact on the software development process, costing money and taking time that could be better spent in adding value for those customers and users.
[ SE-280Dr. Mark L. Hornick
5
Why focus on defects?
Defects have a big impact on the software development process:
Defects must be managed, costing money and taking time that could be better spent in adding value for those customers and users.
With poor quality, there is no time to address other quality issues such as:
• Ease of installation• Safety• Recovery & robustness• Usability & convenience
Generally, defects are more expensive to remove if they are not found until later in the software development process.
Average defect fix time (minutes) [Xerox TSP team]
5
22
2
25
32
1405
1 10 100 1000 10000
Design review
Design insp.
Code review
Code insp.
Unit test
System test
[ SE-280Dr. Mark L. Hornick
7
CMM – Capability Maturity Model
CMM is an assessment of an organization’s software capabilities; an SEI-developed metric with 5 levels:
Level 1 = “no defined process” Level 5 = “process exhibiting continuous
improvement”
[ SE-280Dr. Mark L. Hornick
8
A mature organization at CMM level 5:Creates and reviews requirements
Does design inspections
(design is reviewed by a group)
Does code inspections
(code is reviewed by a group)
Does a lot of testing And still leaves about 1 defect/KLOC
[ SE-280Dr. Mark L. Hornick
9
With the growth of software systems, we are facing a significant quality challenge
What if we want 1-10 defects per MLOC? Level 5 CMMI is about 1000 defects/MLOC!
You probably can't "test in" this quality level
And, probably can't "inspect it in" either: Assume 1000 LOC 30 pages of listings Then, 1 MLOC 30,000 pages Can we inspect 30K pages well enough to leave
only 1-10 defects??
[ SE-280Dr. Mark L. Hornick
10
Quality management (defect removal and prevention) is a critical part of the PSP.
Q. Who injects software defects?A. Individual software engineers!
Fundamental Axiom: The personal process of the individual software engineer is the place to remove defects, determine their causes, and learn to prevent them.
[ SE-280Dr. Mark L. Hornick
11
Remove
Quality Metrics: Yield is a measure of how well we find and remove defects.
Developmentphase
Yield is the % of existing defects we remove in a phase.
Generally, defects are injected in development phases like Design and Coding, but it is possible for
defects to be injected or removed in any phase..
Inject
Developmentphase
RemoveInject
Developmentphase
RemoveInject
Escapes are defects that pass from one phase to the next.
PhaseInjectedInhasesFromPriorP
haseRemovedInPPhase DefectsEscapes
DefectsYield
100
Process Yield is the composite yield for all phases before Test.
CodeReview
Planning
CodeReview
Planning100injected
removed
Defects
DefectsYield
The process yield (or just "yield") target is about 70%; that is, no more than 30% of defects should be left to be found during testing, since testing is expensive!
[ SE-280Dr. Mark L. Hornick
13
There are different kinds of costs associated with quality.
Appraisal costs are the costs of inspecting for defects, (in the PSP, Design and Code review times).
Failure costs result from rework, repair, and scrap • in the PSP, this is time you spend in Test
Both of these metrics are costs of quality; that is,the “price” you pay to ensure the quality of your code.
The relationship between these two kinds of quality costs are described by a ratio:
Failure
Appraisal
COQ
COQFRA /
In the PSP, the A/F ratio should be about 2.0.
A/FR tracks ratio of Appraisal cost (review time) Failure cost (test time)
SE-280Dr. Mark L. Hornick
15
If measured in many organizations, A/FR would be near zero!
but (mature) PSP processes typically exceed 2.0
High A/FR is associated with low test defect density and high product quality
The five PQI components are calculated in the following manner.
OCfectsPerKLUnitTestDe
10PQI
CectsPerKLOCompileDef
20PQI
CodeTime
TimeCodeReviewPQI
DesignTime
ewTimeDesignReviPQI
CodeTime
DesignTimePQI
fectsUnitTestDe
ectsCompileDef
TimeCodeReview
ewTimeDesignRevi
TimeDesignCode
5
10
2
2
Note that all PQI values are limited to a range of 0.0 … 1.0
iOverall PQIPQI
Target value: < 10 defects/KLOC A+M
Target value: < 5 defects/KLOC A+M
Overall PQI is computed by taking the product of the components.
Target: > 0.4
Here is an example of a PQI calculation.
PQIRaw PQI
AdjPQI
DesignCodeTime
DesignReviewTime
CodeReviewTime
CompileDefects
UnitTestDefects
Total PQI
Design Time 125
Design Review Time 38
Code Time 67
Code Review Time 32
Compile Defects/KLOC (A+M) 0
UT Defects/KLOC (A+M) 6.1
2
2
10
5
DesignCodeTime
DesignReviewTime
CodeReviewTime
CompileDefects
UnitTestDefects
DesignTimePQI
CodeTimeDesignReviewTime
PQIDesignTime
CodeReviewTimePQI
CodeTime20
PQICompileDefectsPerKLOC
10PQI
UnitTes
tDefectsPerKLOC
PQIDesignCodeTime
PQICode Review Time
PQICompile DefectsPQIUnit Test Defects
PQIDesign Review Time
Note that Adjusted PQI values are limited to a range of 0.0 … 1.0
Results of the PQI calculations
PQIRaw PQI
AdjPQI
DesignCodeTime 1.87 1.00
DesignReviewTime 0.61 0.61
CodeReviewTime 0.96 0.96
CompileDefects 2.00 1.00
UnitTestDefects 0.90 0.90
Total PQI 0.52
Design Time 125
Design Review Time 38
Code Time 67
Code Review Time 32
Compile Defects/KLOC (A+M) 0
UT Defects/KLOC (A+M) 6.1 OCfectsPerKLUnitTestDe
10PQI
CectsPerKLOCompileDef
20PQI
CodeTime
TimeCodeReviewPQI
DesignTime
ewTimeDesignReviPQI
CodeTime
DesignTimePQI
fectsUnitTestDe
ectsCompileDef
TimeCodeReview
ewTimeDesignRevi
TimeDesignCode
5
10
2
2
PQIDesignCodeTime
PQICode Review Time
PQICompile DefectsPQIUnit Test Defects
PQIDesign Review Time
The process quality index (PQI) is a compact representation of five quality measures.
Text, page 150
The value at each vertex is 1.0 (good), while the center is zero (bad).
TimeDesignCodePQI
ewTimeDesignReviPQI
TimeCodeReviewPQI
ectsCompileDefPQIfectsUnitTestDePQI
The Process Dashboard tool computes a number of quality metrics.
Start looking at your own quality metrics!
What about the use of automated defect-finding tools?
If you have a tool that can remove 100% of a certain type of defect, use it as early as possible to get rid of them!
If you have a tool that can remove about 10% of a certain type of defect, it is not very effective, but also relatively harmless since you won’t rely on it.
If you have a tool that can remove 90% of a certain type of defect, it can be dangerous because you will tend to believe that it found them all, and not aggressively seek out the remaining ones.
[ SE-280Dr. Mark L. Hornick
22
Conclusion
Inspections and Testing will not catch enough defects
It is more much more expensive to inspect and test than to review
Personal Design and Code reviews are key to high quality software