Download - Methods for Estimating Defects
Methods for Estimating Defects
Catherine V. Stringfellow Mathematics and Computer Science Department
New Mexico Highlands University October 20, 2000
Problem and Questions
To improve the efficiency of system testing by helping testers determine when to stop testing and release software.
When should testers stop testing without sacrificing effectiveness?
Can defect estimations be used to make release decisions?
Data
available in defect database release name phase defect reported component name date defect reported and much more
Approach
Estimate number of remaining components with defects using: Capture-Recapture Methods Curve-Fitting Methods Experience-Based Methods
Estimate number of remaining failures using: SRGM Selection Method
Compare estimates to acceptability thresholds to make release decisions
Estimating Components with Defects in Post-Release, but not in Test
Capture/recapture Methods
Curve Fitting Models
Experience-based Estimation Method
(Uses historical data in the estimation)
Capture/Recapture Methods Derived from Wildlife Biological Models Software Defect Estimation Models
Use multiple independent reviewers to count faults (instead of animals)
A reviewer captures a certain fault and other reviewers who identify same fault are said to “recapture it”
Capture/Recapture
x x
x x x
x x x x x x x x x x
x x
xx x
Model requires an overlap if most overlap, few remaining (unless MANY found)
if few overlap, many remaining
Five CRC Estimation Methods M0ML MTML MTChpm (for two reviewers) MHJK MthChao
Models’ AssumptionsModel 1, M0Test Group 1
Test Group 2
Test Group 3
Model 3, MtTest Group 1
Test Group 2
Test Group 3 Detection probabilities
Defect number
Model 2, MhTest Group 1
Test Group 2
Test Group 3 Detection probabilities
Defect number
Model 4, MthTest Group 1
Test Group 2
Test Group 3 Detection probabilities
Defect number
CRC with MtML Example
Sample (CRC with MLE method)n m n1 n2 N >= n L(N)12 2 8 7 12 -6.86
13 -6.5414 -6.4115 -6.3416 -6.32
MAX -> 17 -6.3218 -6.3319 -6.3420 -6.37
Estimated Remaining is 17-12 = 5
Correct remaining = 8
MLE method tends to underestimate
0
2
4
6
8
10
1 2 3 4 5 6 7 8 9 10
Defect #
# d
efe
ct
oc
cu
rre
nc
es
Curve Fitting Models Detection Profile Method (fit with decreasing exp curve)
Cumulative Method (fit with increasing exponential curve)
0
10
20
30
40
1 2 3 4 5 6 7 8 9 10
Defect #
Cu
mu
lati
ve #
de
fec
t o
cc
urr
en
ce
s
Estimated # total defects is smallest x-value with y-value <= 0.5
Estimate is x-value at which expfunction is asymptotic
Sample Datadefect # rev1 rev2 # occurrences Exp Regression
1 1 1 2 1.06762 1 1 2 1.05843 1 1 2 1.03854 1 0 1 1.00005 1 0 1 1.00006 1 0 1 1.00007 1 0 1 1.00008 1 0 1 1.00009 0 1 1 1.0000
10 0 1 1 1.000011 0 1 1 1.0000
n = 12 0 1 1 1.0000
Total n1= 8 n2= 7m = 2 reviewers
Detection Profile Method
y = 1.8196e-0.0654x
0
0.5
1
1.5
2
2.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
defect #
# o
ccu
rren
ces
Look for x-value w here y-value becomes 0.5DPM Estimate approximately 20 total, 20-12=8 remainingCorrect Answ er is 8 remaining
DPM Curve Fitting Example
DPM Estimate: approximately 8 remaining
Correct Answer: 8
Results (3 sites)
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
m0ml mtml mhjk mthChao cumulative dpm (exp) dpm (linear) exp-based
Estimators (3 sites)
Rel
. Err
or
Release 1 Release 2 Release 3 Mean Abs.
4 2 5 8 7 6 3 1Ranks:
SRGMs
Some Models’ Assumptions Test according to operational profile. Failure rate proportional to number of failures
remaining. Defect repair is immediate. Defect repair is perfect. No new code is introduced during the test
period.
SRGMs in practice Assumptions violated.
Previous studies show SRGMs are robust: SRGMs perform well in predicting failure rates.
SRGMs Exponential Model (Musa’s) Delayed S-shaped Gompertz curve Yamada Exponential Yamada Raleigh
Release 1
Release 2
Release 3
SRGM Selection Method
Integrating Estimation Methods
Logical AND: all methods must say stop Sequential
If mhjk says stop, stop. Else if SRGM selection method and one other
method says stop, stop. Else if at least one of the following methods,
m0ml, mtml, dpm(linear), AND the experienced-based method say stop, stop.
Else continue.
Integrating Estimation Methods Majority
Group m0ml, mtml, dpm(linear) together. If one says stop, stop.
In case of tie: a) continue test for another week; or b) compare number of defects in last week to acceptability threshold (5)