using bayesian nets to predict software defects in arbitrary software lifecycles martin neil agena...

Using Bayesian Nets to Predict Software Defects in Arbitrary

Software Lifecycles

Martin NeilAgena Ltd

London, UK

Web: www.agenarisk.comEmail: [email protected]

Telephone: +44 (20) 7404 9722

Effective risk assessment in software projects

• See problems early and take mitigating actions

• Predict when project will be late and/or over-budget

• Decide early enough if project is infeasible

• Predict product quality and the impact of poor reliability

• Know when it makes sense to stop testing

• Support process improvement by determining the impact of good/poor processes

Impediments to effective software risk measurement

• Lack of data• Difficulty of communicating risks• Poor requirements management• Poor coordination with stakeholders and

customers• Difficulty in managing expectations• Difficulty of making progress visible• Lack of applicability of traditional project

management tools

New solutions for software project risk

• Risk maps for high-level project risk assessment

• Risk maps for managing and predicting defects during and after development

• Risk maps for project risk health checks

What the defect prediction risk maps provide

• Accurate predictions of defects at different phases

• Know when to stop testing• Identify where to allocate testing• Minimise the cost of rework• Minimise danger and cost to user• Highlight areas for improvement• Approach fully customisable• Models arbitrary project life-cycles

AgenaRisk modelling and validation

• AgenaRisk template risk maps designed using– Expert judgement– Published data– Company data– Published models

• Risk maps validated on projects at:– QinetiQ– Israel Aircraft Industries– Philips Netherlands and India

What’s special about the approach?

• Structured Method (for quantifying ALL risks)– Based on well proven Bayesian technology– Beyond current statistical & Monte Carlo techniques– Combines subjective judgements with historical data

• Visual Communication (of complex risk issues)– Manages risk through pictures– Useable by non-specialists as well as risk experts– Makes analysis of complex risks easy to communicate

• Robust (for large & complex risk applications)– Enables scalable, reusable & auditable risk models– Integrates easily with databases and spreadsheet systems– Enables developers to build end-user applications

Traditional metrics and statistical approaches

• Software metrics studies provide useful empirical baseline

• Regression-based models are good for predicting the past

• .. but are of little use for true risk assessment

Background: History of Reliability Modeling at Philips - Bangalore

Study of Reliability Models

AID

2000

MODIST

Agena Risk

2001 20032002 2004

Phase-level risk map for defect prediction

Schematic of (single) phase-level risk map for defect prediction

Newfunctionality implemented

Spec & docProcessquality

DevelopmentProcessquality

TestingProcessquality

ReworkProcessquality

DefectsPRE

Totaldefects in

DefectsPOST

Defects found

Defects fixed

Running the defects risk map






DefectsPRE

Totaldefects in

DefectsPOST

Defects found

Defects fixed

250

0

Median value 257

High quality development






DefectsPRE

Totaldefects in

DefectsPOST

Defects found

Defects fixed

150

0

Very highVery high

Median value 40

Median value 25

… but suppose more than expected defects found in testing






DefectsPRE

Totaldefects in

DefectsPOST

Defects found

Defects fixed

150

0

Very highVery high

Median value 204

400

So process assumptions “overruled”by hard facts from QA/test results

How good must your testing and rework be to get ZERO residual defects?






DefectsPRE

Totaldefects in

DefectsPOST

Defects found

Defects fixed

150

0

0

Linking phase level risk maps in AgenaRisk

Enables us to modelarbitrary projectsand process

Level of granularity can be chosen to suit the availabledata

AgenaRisk Validation Objectives

• To evaluate AgenaRisk (Method+Toolset) for its

defect prediction capabilities in software projects

• To help Philips decide whether it is a potential

candidate for deployment within software

development centers of Philips CE

• Validation was conducted during Q4/2004 -

Q1/2005

Distribution of selected projects

MM&C (9)28%

TV (16)50%

DVD (7)22%

(actual number of projects shown in brackets)

Results: Actual vs Predicted Defects (Round 1, 2 & 3)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

# o

f de

fect

s

Actual 672 680 928 736 546 209 476 204 196 653 4223 688 261 1597 71 1906 148 228 616 253 53 1768 31 412 373 881 167 109 91 17 5 33

P redicted Rnd 1 666 722 695 759 181 350 551 274 164 467 2115 338 337 350 217 910 271 380 1264 447 146 447 231 430 349 721 246 237 235 200 260 200

P redicted Rnd 2 674 722 986 854 641 254 581 262 259 505 3190 517 407 1147 117 1307 278 485 1419 437 135 976 168 201 180 125

P redicted Rnd 3 2509 444 1514 51 1886 75 355 1272 501 48 1526 52 123 145 116 57 46 55

CEA2UIR

E

CE96DARE

CE11EU2

K

CEK6HDDL

CEC8UIF

T

CED6CHGR

CEL9CEMW

CEK9P SMW

CE39DARE

CEC9CSVC

CEI7FOC

E

CEE8CSVC

CEM4LC04

CEC6UI0

2

CEA540SA

CEH6UIA

2

CEN2DVP 9

CEH2LC3S

CEE3DRCS

CE44A02C

CED7963

S

CEE7UIL

S

CER7DIS

Y

CEO5P CP S

CEI6P I03

CEK2P CSW

CEM9

P SA

CEE7UILS-

CEK5P 2CL

CEQ1RC94

CEP3P EV3

CEN3NAKA

Actuals Vs Predictions

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 500 1000 1500 2000 2500

Actual defects

pre

dic

ted

def

ects

Validation Summary

• Approach perceived as innovative and promising

• Our evaluation showed excellent results for project sizes between 10 and 90 KLOC

• Initially, projects outside this range were not within the scope of the default model so predictions were less accurate, but accuracy was significantly improved after tailoring standard model

The Road Ahead

• Revised model tailored as Philips-specific for even more accurate results and for projects outside the scope of the default model– New structure already proposed– Replacing the underlying assumptions (where required)– Tailoring questionnaire interface for end-user (with explanations

using organization-specific terms)– Training potential “users” for “what-if-scenarios” and “risk

assessment”• Used AgenaRisk on live projects• Use AgenaRisk for post-release (Field Call Rate) defect

prediction• Incorporate other factors like effort, schedule, etc

Summary

• New approach to defect prediction and quality control based on Bayesian Networks, enables us to– Take account of process and people factors (with support for

process improvement)– Monitor and predict fault injection-detection and rework through

lifecycle– Avoid project failures and improve project communication – With minimal additional data collection requirements and costs

• Validation results show that the AgenaRisk tool and models can be used as is and that simple tailoring can enable even greater power and accuracy for Philips

using bayesian nets to predict software defects in arbitrary software lifecycles martin neil agena...

Documents

development risk maps

effective risk assessment

project risk health

true risk assessmentbackground

software project riskrisk

software defects

software projectssee

bayesian nets