08 economics of testing (v2.5)

© SIM Group Ltd., SQS Group AG, 2002

ISEB Foundation Certificate in Software Testing

Economics of TestingEconomics of Testing


Economics of Testing

In this session we willIn this session we will

Look at the cost of testing (or the cost of not testing)Look at the cost of testing (or the cost of not testing)

Appreciate the value of good, timely testingAppreciate the value of good, timely testing



The earlier a fault is found, the cheaper it is to remedyThe earlier a fault is found, the cheaper it is to remedy

Faults in the requirements can lead to major re-engineering of Faults in the requirements can lead to major re-engineering of the entire systemthe entire system

Many faults can be found using reviewsMany faults can be found using reviews Reviews of documentation and / or codeReviews of documentation and / or code

Allows testing to find more ‘substantial’ faultsAllows testing to find more ‘substantial’ faults



The average cost of fixing a defect increases tenfold The average cost of fixing a defect increases tenfold with every step of the development processwith every step of the development process

In code reviews you will find and fix defects in an average of In code reviews you will find and fix defects in an average of 1 - 2 minutes1 - 2 minutes



In initial testing, defect fix times will average between In initial testing, defect fix times will average between 10 to 20 minutes10 to 20 minutes

In integration testing each defect can cost an hour or In integration testing each defect can cost an hour or moremore

In system test each defect can cost 10 to 40 or more In system test each defect can cost 10 to 40 or more engineer hoursengineer hours

Watts Humphrey Watts Humphrey



Early test design can prevent fault multiplicationEarly test design can prevent fault multiplication

Analysis of specification during test preparation often brings Analysis of specification during test preparation often brings faults in the specification to lightfaults in the specification to light

If faults in documentation are not found then the system may If faults in documentation are not found then the system may be developed incorrectlybe developed incorrectly


Economics of testing

An exampleAn example



Finding the error in the requirementsFinding the error in the requirements

Change requirements documentChange requirements document



Finding the error in the specificationsFinding the error in the specifications


Change specificationsChange specifications



Finding the error in the codeFinding the error in the code



Change the codeChange the code



Finding the error in system testingFinding the error in system testing




Change the tests, re-test & regression testChange the tests, re-test & regression test



Finding the error in UATFinding the error in UAT





Re-do UATRe-do UAT



Finding the error in LiveFinding the error in Live





Re-do UATRe-do UAT

Plus...Plus...



Finding the error in LiveFinding the error in Live

User to report and log faultUser to report and log fault

Users must be advised of fix, work-around etcUsers must be advised of fix, work-around etc

The live database may need to be fixedThe live database may need to be fixed

Deploying the new build may take hoursDeploying the new build may take hours

Loss of businessLoss of business



AdditionallyAdditionally

Faults found in coding or later require changes to the code Faults found in coding or later require changes to the code librarylibrary

Documentation and help files will have to be changedDocumentation and help files will have to be changed

Every fault requires investigating and loggingEvery fault requires investigating and logging

Administration costsAdministration costs



The cost of not TestingThe cost of not Testing

What is the cost of testing?What is the cost of testing?

Which is cheaper? Which is cheaper? Testing and finding faults before releaseTesting and finding faults before release Not testing and finding faults once the system is liveNot testing and finding faults once the system is live

Companies do not typically have figures for eitherCompanies do not typically have figures for either



SummarySummary

The purpose of testing is to find faultsThe purpose of testing is to find faults

The earlier a fault is found the cheaper (and easier) it is to fixThe earlier a fault is found the cheaper (and easier) it is to fix

Starting testing early will help find faults quickerStarting testing early will help find faults quicker

08 economics of testing (v2.5)

Documents