testing and the bottom line: a new method to estimate the value of good software testing' by...
TRANSCRIPT
© 2011 Codenomicon Ltd. 1
Testing and the Bottom Line
A New Method to Estimate the Value of Good Software Testing
Juha-Matti Tirilä
Codenomicon Ltd.
Outline
• What are the problems in estimating the cost of testing
• What happens when code is tested well: the effect on bugfix price of early discovery
• Our method for estimating the cost impacts of software testing
Why develop a new cost model
• Problem: the cost structure of testing is typically misunderstood
• The difference in nature between testing and development– Cost of testing cannot be analyzed with the same
tools as the cost of development
Cost of testing
• Indirect cost: if we do not test properly, how much is it going to cost us
• Direct cost: the price of performing the testing – Plus the cost of fixing the issues? It depends.
Why does "it depend"?
• You may have found the bug otherwise just as well, only now you find it earlier, so there’s no extra cost
• Even though it costs you money, testing will have positive consequences beyond the particular bug– (That are very difficult to measure)
Cost-benefits of early bug discovery
• Especially in a security context: vulnerabilities detected by third parties tend to be expensive due to – Negative reputation– Downtime– Increased need for customer support– Etc...
Cost-benefits of early defect discovery
• Thinking of security critical bugs: – Post-release vs. pre-release– Impact on accountability – Flexibility in resource allocation; no need to fix
straight away
Cost-benefits of early defect discovery
– The person who wrote the code in the first place still working
– Maybe even still remembers the logic of the code in question
– Organization-level practices in maintaining good quality, improved performance in the long run
Cost-benefits of early defect discovery
– Especially in security testing, it is a rather limited set of bugs that appear again and again so that testing: • Improves your knowhow• Accelerates production speed• With the best tools:
– very to-the-point reports on what types of bugs were discovered
– easy to fix bugs– educate developers to avoid similar mistakes in the future
The model explained
• Product release at T = 6 months• The price of bugs increases until T, then drops
a little, starts to rapidly rise immediately afterwards
• Some bugs cost nothing• The effect of testing: ”all” bugs discovered
earlier so that the distribution tends to the left, and down
Comparison to traditional incident probability calculations
– One approach: try to determine the probability and cost of impact, and evaluate your testing budget against this expected loss
– Our approach: more statistically oriented, more geared towards estimating the average savings irrespective of whether issues ever surface or not
But you never have the data before it’s too late, right?
• Remark: the dual presentation done above is something you in reality never get. You have a certain level of testing, and you don't know what the other alternatives would have cost you. How to deal with this problem?
• Solution: use statistical distributions and models, and tune the parameters of the model to reflect your development environment.
A peek at the parameters
– Agree upon a trend for how the cost of a bug fix will develop towards release, and after release
– Agree upon the way testing is going to affect discovery times of issues
– Agree upon "variance", i.e. how widely the cost per delivery time is distributed
Benefits of using distributions
• Very little immediate effect on the estimated price of fixing bugs
BUT• Provides lots of additional information• The mechanism is easy to understand and
tune to match a particular development environment
In the long run, better estimates to rely on
Benefits of using distributions
• You can readily calculate various statistics, such as: – The expected cost of fixing an average bug, with
and without testing– The expected cost of fixing bugs detected in
pre/post -development phases– Etc... , to your liking, any statistics computable
from a statistical distribution – So for example also estimate risks
Challenges
• Of course, the quality of all the estimates depends on how well you tune the parameters
But:
• You can use real-life data to estimate the parameters