UK NDA Application of

Reference Class Forecasting

in Optimism Bias Adjustment Karl Sanderson

Head of Cross-Industry Learning

3 October 2019

NDA Historical Context

• 2012 report by the

National Audit Office

detailed historical

problems with cost &

schedule estimates at

the NDA - it notes that no

OBA was applied to past

estimates and that

contingency was

inadequate

• 2018 update report noted

that overall lifetime cost

estimating had improved

but large uncertainties

still remained

3

Reference: Amyas Morse., The Nuclear Decommissioning Authority:

Progress with Reducing Risk at Sellafield, London, National Audit Office,

2018

Optimism Bias Adjustment

• Optimism Bias (OB) defined in HM Treasury Green Book

supplementary guidance as: “a demonstrated, systematic,

tendency for project appraisers to be overly optimistic”

• Observable trends in underestimated project cost and

schedule projections are attributable to OB - these trends

are present in academic research, industry data, and

analysis of NDA projects

• As mitigation, Optimism Bias Adjustments (OBA) can be

applied to project cost & schedule estimates,

compensating for lack of understanding of project risks

early in the lifecycle

• Reference Class Forecasting (RFC) is the technique used

to determine the NDA-specific OBA 1

OBA relationship to other Estimation Methodologies

Reference Class Forecasting on its own does not deliver

better quality estimates. OBA needs to be used alongside;

• More effective Risk Management (threat & opportunity

identification) and differentiation from uncertainties and

ambiguities

• More effective probabilistic analysis / Quantitative Risk

Analysis (QRA)

• The use of better / more accurate volumetric /

Benchmarking data

• Improved Project Management in delivery

• More effective and robust challenge (Assurance)

2

Quantitative Risk Assessment

– example contributory factors

5

Procurement Project

Specific

Client

Specific Environment

External

Influence

• Complexity of Contract

Structure

• Late Contractor

Involvement in Design

• Poor Contractor

Capabilities

• Government Guidelines

• Dispute and Claims

Occurred

• Information management

• Other (specify)

• Design Complexity

• Degree of Innovation

• Environmental Impact

• Other (specify)

• Inadequacy of the

Business Case

• Large Number of

Stakeholders

• Funding Availability

• Project Management

Team

• Poor Project Intelligence

• Other (specify)

• Public Relations

• Site Characteristics

• Permits / Consents /

Approvals

• Other (specify)

• Political

• Economic

• Legislation / Regulations

• Technology

• Other (specify)

Quantitative

Risk Assessment

Uncertainty + Quantified Risk (Impact / Likelihood)

Monte Carlo Analysis

… not an exhaustive list of

threats and opportunities

Methodology at Early Stage Gates

4

Quantitative Risk Assessment

Reference Class Forecasting & OBA

OBC SOC FBC

Base Estimate

Assurance, Benchmarking, Project & Risk Management Estim

ate

Maturity

FEED / Detailed Design Preliminary Design Scoping

Single Concept Short-listed Concepts All Concepts

Single

Concept

Selected

Contract

Award

Then

Execution

HMT Green Book OBA

• HM Treasury Green Book guidelines based on generic project

categories:

4

Literature Data

5

Budzier et al looks at

a reference class of

216 projects to better

understand the

chances of cost

overruns in nuclear

projects

The sample included

194 nuclear new

builds and 22 nuclear

waste storage

projects

This data shows a

20% chance of going

over budget by over

202% Reference: Budzier et al. Quantitative Cost and Schedule Risk Analysis

of Nuclear Waste Storage, Oxford Global Projects, 2018

NDA Data Sample and Analysis

• Data was obtained covering a representative sample of

Sellafield projects including:

• Cost data on 22 projects covering different lifecycle points (14

covering the whole lifecycle)

• Schedule data on 9 projects covering the whole lifecycle

• For each project an initial and final best estimate was

determined; typically this was a P50 or “most likely” value

from a range - these were used to determine the

percentage increase over the project’s lifetime

• This data was used to plot S-curve diagrams showing the

percentage uplift associated with a given acceptable

chance of overrun (shown in the following slides)

6

Sample Data

7

6

2

3

1

2

0

1

2

3

4

5

6

7

8

9

10

<50% 50% to 120% 120% to 190% 190% to 260% 260% to 330%

Number of

Projects

Histogram Illustrating the Change in Cost Estimates for Projects at Sellafield

Results – Entire Project Lifecycle (from SOC)

8

Results – Preliminary Design Gate (from OBC)

9

Results – Detailed Design Gate (from FBC)

10

Results - Schedule Analysis

11

• The same analysis was carried out using data on schedule

estimates. Less data was available here resulting a

smaller sample of 9 projects covering the whole lifecycle.

Overall Results

• The results from the NDA sample data are as follows:

• Compares closely to the results from the larger global

sample presented in Budzier et al, 2018

• … suggests that the larger sample of global nuclear

projects might be representative of projects undertaken

within the NDA group

12

Chance of Overrun: 50% 20%

Cost 76% 195%

Schedule 65% 111%

Chance of Overrun: 50% 20%

Cost 67% 202%

Schedule 40% 104%

At SOC

16

Build estimate using best

available methodology

and best available data

Determine relevant

reference class for

project

Apply reference class

‘uplift’ using P50

confidence to the

estimate (optimism bias

adjustment)

Apply reference class

‘uplift’ using P80

confidence to calculate

an upper estimate Quote AFC / AFS estimate

range based on the base

estimate and optimism bias

adjustment

Estimating Reference

Class1 or OBA2 QRA Report

Baseline

Estimate

Preliminary Cost

/ Schedule

Range

Notes

1. NDA OBA

2. HMT OBA

At OBC & FBC

17

Update estimate in line

with scope

Determine relevant

reference class for

project

Apply reference class

‘uplift’ using P50

confidence to the

estimate

Quote AFC / AFS range

(P50-P80) based on the

QRA

Estimating Reference

Class QRA Report

Baseline

Estimate

P50 and P80 Cost

and Schedule

Perform QRA, taking into

account the project,

programme, wider risk

factors to derive P50.

Compare QRA at P50

with reference class at

P50

Is QRA consistent

with reference class

NO

YES

Conclusions & Next Steps

When a Project Manager proposed an early stage base estimate (at SOC

equivalent, with allowances but no contingency), consider adjusting for

optimism bias as follows:

Cost For P50 add 67-76%

For P80 add 195-202% (triple it)

Schedule For P50 add 40-65%

For P80 add 104-111% (double it)

Going forward:

• a wider sample of global nuclear projects, and NDA estate projects

will be used to derive specific results for the NDA

• further academic research, for example on statistical validity of

reference classes

13

Recap

8

OBC SOC FBC

Application of

RCF only

QRA tested by RCF QRA tested by RCF

Reference Class

S- curves used to

replace the

deterministic HMG

Green Book factors

(upper bounds) -

based on relevant

experience

OBA OBA OBA

££

Probability

Risks and uncertainty Risks and uncertainty

££

Probability

Estimate

Estimate

OBA

Estimate Estimate

QRA QRA