Consequences of Mispredictions of Software Reliability

Rakesh RanaUniversity of Gothenburg,

Sweden

rakesh.rana@gu.se

Predicting Software Reliability

Tracking and predicting quality challenge.

Software defects observable and useful indicator to track and forecast software reliability.

Software reliability measures are primarily used for [1]:• Planning and controlling testing resources

allocation, and• Evaluating the maturity or release readiness.

[1] C.-Y. Huang, M. R. Lyu, and S.-Y. Kuo, “A unified scheme of some nonhomogenous poisson process models for software reliability estimation,” IEEE Trans. Softw. Eng., vol. 29, no. 3, pp. 261–269, 2003.

Predicting Software Reliability

Software Reliability (SR): is (A) the probability that software will not cause the failure of a system for a specified time under specified conditions. Or (B) the ability of a program to perform a required function under stated conditions for a stated period of time.

Software Reliability Model (SRM) is a mathematical expression that specifies the general form of the software failure process as a function of factors such as fault introduction, fault removal, and the operational environment.

IEEE 1633: Recommended practice on software reliability

SRGMs: Software Reliability Growth Models

Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif

Research Question

Given the software quality growth prediction curve, what are the consequences of mispredicting the total number of defects and release readiness?

we explicitly recognize two common axes of the accuracy of predictions – (i) the prediction of the asymptote, and (ii) the prediction when the total number of defects is

discovered

Mispredicting the asymptote

Over-predictions• Too high expectations – more pressure on the testing team. • Assumption that testing is ineffective. • Additional cost of test analyses in search for new test areas (unnecessary).• Risk for postponing release.• Risk for lost time to market.• Risk for wasted costs for testing.• Risk for unnecessary RCAs to find area which are not tested enough.

Mispredicting the asymptote

Under-predictions• Releasing the product with defects.• Additional costs for post-release defect removal activities and patches.• Defects which are manifested as integration problems requiring quick

fixes.• De-prioritizing testing effort at early stages and thus finding large number

of late (and thus costly) defects during system/acceptance testing.

Mispredicting release readiness

Early-predictions• Releasing the software with defects.• Higher cost of corrective maintenance of the product.• Postponing the release (if the mispredictions are discovered before the

release).

Mispredicting release readiness

Late-predictions• Unnecessary additional testing resources to get back on track• Postponing the release in expectation of more defects to come.• Additional costs of test analysis to increase the speed and effectiveness of

testing.

Mispredicting the shape of the curve

Actual shape Expected shape Concave S-shaped Convex

Concave

Over-prediction of the total number of defects

Over-prediction of the total number of defects

S-shaped

Release readiness is predicted too early

X% of found defects is predicted earlier than expected

Over-prediction of the total number of defects

Convex

Release readiness is predicted too early

X% of found defects is predicted earlier than expected

Too much resources for late testing

Industrial Validation

Application domain

Software development process

Current methods for software defect prediction

Automotive

V-shaped software development mostly

using sub-suppliers for implementation

Focus on status visualization and analogy

based prediction

Telecom Agile development,

mostly in-house

Various modes of presenting current status and predictions methods

VCG, Volvo Cars Group

• A number of different metrics are collected and monitored continuously

• Forecasts are used to track release readiness.

• Root cause analysis• focus - what can be done now to get on track?

• It is highly important to meet the release dates, more resources get mobilized and allocated where needed.

• Under-predictions: Setup task force (resource mobilization). • Not seen as major problem (if limited to a few ECUs), potential

problem if widespread across platform (large project).

• Over-prediction: is not seen as a critical problem.

VCG, Volvo Cars Group

• Early-predictions: No impact if the project is small - as risk can be easily managed at any stage of project.

• For larger platform projects, the forecasts will be re-checked consecutively for a period of time and cross-validated by different expert opinions before resources are planned according to forecasts.

• Late-predictions: Strategy to find areas affected by late-predictions.

• The test resource would be balanced in light of new information and with the aim to meet quality requirements by the release date.

Ericsson

The impact of mispredictions have two dimensions – (i) metric team which delivers the predictions and (ii) project where the predictions are used.

For the metrics team:• All mispredictions make the team lose trust from the organization.

• Once the organization acts upon wrong predictions the team loses the ability to influence – the next time the organization will need a second opinion before acting.

• This increases the cost of predictions in the long run.

Ericsson

For the projects:

Over-predictions:• Strengthening and reallocation of resources – if this is done during a

long period of time then this impacts the release date negatively

Under-predictions:• Negative impact on the release date

• Ordered overtime/extra resources – when the organization finds that the reliability was under-predicted

• Reallocation of resources – when the organization finds that the reliability was under-predicted.

Conclusions

Research objectives

Given the software quality growth prediction curve, what are the consequences of mispredicting the total number of defects and release readiness?

Strategies to avoid mispredictions:

Predict often Experiment with three types of curves Predict the shape of defect inflow using available data