Methods for Estimating Defects

Catherine V. Stringfellow Mathematics and Computer Science Department

New Mexico Highlands University October 20, 2000

Problem and Questions

To improve the efficiency of system testing by helping testers determine when to stop testing and release software.

When should testers stop testing without sacrificing effectiveness?

Can defect estimations be used to make release decisions?

Data

available in defect database release name phase defect reported component name date defect reported and much more

Approach

Estimate number of remaining components with defects using: Capture-Recapture Methods Curve-Fitting Methods Experience-Based Methods

Estimate number of remaining failures using: SRGM Selection Method

Compare estimates to acceptability thresholds to make release decisions

Estimating Components with Defects in Post-Release, but not in Test

Capture/recapture Methods

Curve Fitting Models

Experience-based Estimation Method

(Uses historical data in the estimation)

Capture/Recapture Methods Derived from Wildlife Biological Models Software Defect Estimation Models

Use multiple independent reviewers to count faults (instead of animals)

A reviewer captures a certain fault and other reviewers who identify same fault are said to “recapture it”

Capture/Recapture

x x

x x x

x x x x x x x x x x

x x

xx x

Model requires an overlap if most overlap, few remaining (unless MANY found)

if few overlap, many remaining

Five CRC Estimation Methods M0ML MTML MTChpm (for two reviewers) MHJK MthChao

Models’ AssumptionsModel 1, M0Test Group 1

Test Group 2

Test Group 3

Model 3, MtTest Group 1

Test Group 2

Test Group 3 Detection probabilities

Defect number

Model 2, MhTest Group 1

Test Group 2

Test Group 3 Detection probabilities

Defect number

Model 4, MthTest Group 1

Test Group 2

Test Group 3 Detection probabilities

Defect number

CRC with MtML Example

Sample (CRC with MLE method)n m n1 n2 N >= n L(N)12 2 8 7 12 -6.86

13 -6.5414 -6.4115 -6.3416 -6.32

MAX -> 17 -6.3218 -6.3319 -6.3420 -6.37

Estimated Remaining is 17-12 = 5

Correct remaining = 8

MLE method tends to underestimate

0

2

4

6

8

10

1 2 3 4 5 6 7 8 9 10

Defect #

# d

efe

ct

oc

cu

rre

nc

es

Curve Fitting Models Detection Profile Method (fit with decreasing exp curve)

Cumulative Method (fit with increasing exponential curve)

0

10

20

30

40

1 2 3 4 5 6 7 8 9 10

Defect #

Cu

mu

lati

ve #

de

fec

t o

cc

urr

en

ce

s

Estimated # total defects is smallest x-value with y-value <= 0.5

Estimate is x-value at which expfunction is asymptotic

Sample Datadefect # rev1 rev2 # occurrences Exp Regression

1 1 1 2 1.06762 1 1 2 1.05843 1 1 2 1.03854 1 0 1 1.00005 1 0 1 1.00006 1 0 1 1.00007 1 0 1 1.00008 1 0 1 1.00009 0 1 1 1.0000

10 0 1 1 1.000011 0 1 1 1.0000

n = 12 0 1 1 1.0000

Total n1= 8 n2= 7m = 2 reviewers

Detection Profile Method

y = 1.8196e-0.0654x

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

defect #

# o

ccu

rren

ces

Look for x-value w here y-value becomes 0.5DPM Estimate approximately 20 total, 20-12=8 remainingCorrect Answ er is 8 remaining

DPM Curve Fitting Example

DPM Estimate: approximately 8 remaining

Correct Answer: 8

Results (3 sites)

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

2

m0ml mtml mhjk mthChao cumulative dpm (exp) dpm (linear) exp-based

Estimators (3 sites)

Rel

. Err

or

Release 1 Release 2 Release 3 Mean Abs.

4 2 5 8 7 6 3 1Ranks:

SRGMs

Some Models’ Assumptions Test according to operational profile. Failure rate proportional to number of failures

remaining. Defect repair is immediate. Defect repair is perfect. No new code is introduced during the test

period.

SRGMs in practice Assumptions violated.

Previous studies show SRGMs are robust: SRGMs perform well in predicting failure rates.

SRGMs Exponential Model (Musa’s) Delayed S-shaped Gompertz curve Yamada Exponential Yamada Raleigh

Release 1

Release 2

Release 3

SRGM Selection Method

Integrating Estimation Methods

Logical AND: all methods must say stop Sequential

If mhjk says stop, stop. Else if SRGM selection method and one other

method says stop, stop. Else if at least one of the following methods,

m0ml, mtml, dpm(linear), AND the experienced-based method say stop, stop.

Else continue.

Integrating Estimation Methods Majority

Group m0ml, mtml, dpm(linear) together. If one says stop, stop.

In case of tie: a) continue test for another week; or b) compare number of defects in last week to acceptability threshold (5)