uncertainty, decisions, models & people - ausimm · industry performance: ... portfolio...

GeoMet 2013, Brisbane Carrasco Lecture: Uncertainty, decisions models & people Steve Begg

GeoMET 2013 Carrasco Lecture, 30Sept2013

Uncertainty, Decisions, Models & People

Steve BeggAustralian School of

PetroleumUniversity of Adelaide


Outline

• Uncertainty, Decisions & Modeling– Uncertainty & Business Perfomance– Uncertainty & Decision-making– Need for integrated models

• Uncertainty, Judgments & People– The Nature of Uncertainty – Uncertainty & Value Maximization


Industry Performance: Comments & Observations

• Super-major:– “Every one of our 10 most important projects failed to

generate the desired return.”

• Large independent:– “The actual performance of our key assets wasn’t even within

the P1 to P99 range.”

• CEO to manager:– “I want your guarantee that we will not spend more than the

P50 on this project!”

• IPA:– “The bigger and more important a project gets, the more likely

it ends up in the “disaster” category (1 in 8 major projects)”


Uncertainty

A fundamental problem

Business outcomes not living up to expectations or possibilities!



Business outcomes not living up to expectations or possibilities!

• People tend to grossly under-estimate uncertainty– number of uncertain factors and the magnitude of uncertainty and is

its consequences (good or bad)– complexity of the relationships between them and therefore un-

anticipated non-intuitive outcomes)

• Naive understanding of NPV “rule” – Uncertainty (and/or delay) = Value Loss. Biased to risk mitigation.

• Better decision-making requires accurate (= unbiased & appropriate range) uncertainty assessment & response– Reduce uncertainty only IF it can change a decision AND expected

benefit of reduction is less than its cost– Mitigate downside risk AND capture upside opportunities– Exploit interactions – dependencies, correlations


Uncertainties matter: they are everywhere in the (Oil & Gas) evaluation “system” ……

EconomicsAsset A

. . .

OOIP Model

Production Data

ReservoirSimulation

Petro-physics

Portfolio

EconomicsAsset 1

EconomicsAsset n

Prices

Taxes

PredictedProduction

Royalties/PSC

Decline

Seismic

Geology

Drilling

OpEx

CapEx

ProductionAllocation

Export

Processing


Uncertainties matter: they are everywhere in the evaluation “system” ……

EconomicsAsset A

. . .

Resource Model

Production Data

ResourceSimulation

Petro-physics

Portfolio

EconomicsAsset 1

EconomicsAsset n

Prices

Taxes

PredictedProduction

Royalties/PSC

Depletion

Geophysics

Geology

Mining

OpEx

CapEx

Export

Processing


….. and occur at many levels …….

• Each domain must address nested layers/types of uncertainty, e.g. sub-surface description, …...

Data

Spatial/Temporal variability

Parameter / TI

Interpretation/Model

Monte Carlo

Geostatistics (or mp)

Experimental Design

Scenario Modeling, Discrete Probabilities

…... with appropriate modeling techniques


…. and we spend a lot of money without really knowing which ones matter

Average Porosity

Rig Cost

Recovery Factor

Gross Rock Volume

Fiscal Terms

Net:Gross

Saturation

Continuity

Oil Price

Facilities

UncertaintyParameter Impact on NPV

Take more core

Renegotiate contract

Build simulator model

Buy more seismic

Fire lawyers

More gamma logs

Different rock model

Survey Analogues

Hedge with futures

Cheaper Steel Supply

Action

"There is nothing so inefficient as very efficiently doing the wrong things".

Peter Drucker


…. and we spend a lot of money without really knowing which ones matter

Average Porosity

Mining Cost

Recovery Factor

Gross Ore-body Volume

Fiscal Terms

Net:Gross

Grade

Continuity

Copper Price

Processing Facilities Cost


Take more core



Drill more holes

Fire lawyers

More gamma logs


Survey Analogues

Hedge with futures


Action




Better uncertaintymanagement

BetterDecisions

Betterperformance

Flawed Decision-Making


Decision-making is about ranking:Case 1: we don’t even need a decision criterion

A

Unc

erta

inty

in d

ecis

ion

Crit

erio

n, e

gN

PV

B DCAlternatives (choices)

We only need to predict values precisely enough to determine which is the best alternative

Here, B is clearly the best choice


Decision-making is about ranking:Case 2: need a decision criterion

A B DCAlternatives (choices)

Unc

erta

inty

in d

ecis

ion

Crit

erio

n, e

gN

PV

We only need to predict values precisely enough to determine which is the best alternative


Decision Criteria

• OK, so we’ve got our PDFs of some decision variable (egNPV, tonnage) for each decision alternative

• What number should we use as a decision criterion?– the P10, P50, P90?– the mode?– the mean? NPV

0

• Answer: Expected Value (for a “risk-neutral” decision-maker)- BUT don’t “expect” the Expected Value!

• By “decision criterion”, we mean a number that can be used– to compare the different decision alternatives, so that we can

choose the best (if alternatives are mutually exclusive)– or compare against a minimum acceptable hurdle (if there is only

one alternative, or multiple non-mutually exclusive alternatives)


Decision Criterion under Uncertainty: Choose the alternative with maximum Expected Value

• Intuitively– Defines what would happen “on average” if we repeat the situation

numerous times

• Mathematically– The probability weighted average of the possible values (discrete PDF)

n

Expected Value = pi xi1

• BUT, don’t “expect” the Expected Value!• No other metric (Mode, P10/50/90, etc) will give a higher

total value over multiple (different) decisions• For a “risk-neutral” decision-maker, it is value they would

attribute to the alternative if all uncertainty was removed.


Avoiding Confusion: Decision Criteria v Predictions

• The PDF is the prediction of all possible outcomes (and their associated probabilities)

• The mode is the Most Likely outcome - So, perhaps, it is the best prediction of outcome

• The Expected Value is a Decision Criterion – ie used to make the best decision – it is not a prediction of the outcome!- at least, it is no more of a prediction than any of the other possible

outcomes in the PDF are- it might not even be a possible outcome!

• For decision making we need unbiased estimates of the EVs of our decision criteria (eg NPV, Reserves)- which requires propagating (unbiased) assessments of uncertainty

in input variables through to uncertainty of the decision criteria


Technical Work in the context of Decisions and Uncertainty: Knowing when enough is enough.

• Technical work is fundamentally about uncertainty assessment for the purpose of making decisions

– First priority: Accurate (=unbiased) uncertainty assessment – Second priority: Uncertainty reduction.

• But if you have a “make the best possible prediction” focus, there is no stopping rule

– you can always reduce uncertainty a bit more (more data, more time, more detail, more accurate physics/geology/tax, more analysis, ...)

• A decision-driven (ranking) focus gives a trivially simple stopping rule

– Stop when further analysis doesn’t change the decision!!

The main role of a geoscientist, engineer or economist is to support decision-making


Propagating Uncertainty:Flaw of Using Averages (after Savage)

Log, Price,NCF...


Averages don’t always work

• For non-linear processes,– reservoir simulation– volumetrics with cut-offs– development alternatives

even if only a single, “best” estimate is required, we still need to use complete range of inputs - cannot use an average input

• Also, P10 (P90) results are NOT given by taking P10 (P90) inputs and running them through the model

ResultY = 4

ModelY = X2 Z2/

3 21 1 8

Mean = 10 Mean = 5

x z

Y

Simulation Result

0 30

True Mean~ 7.8


The Flaw of Averages: implications for decision metrics

• If f is a decision metric (eg NPV) computed from a model with uncertain input variables x,y,z,…

unless f is linear

, ,[ ( ]....)x y zf

( [ ], [ ], [ ], )...x y zf


On Models ……

“…. when you can measure what you are speaking about and express it in numbers, you know something about it. But when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind. It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science, whatever the matter may be.”

Lord KelvinTaken from Davis, 1986


On Models ……

What could be cuterThan to feed a computerWith wrong informationBut naïve expectationTo obtain with precisionA Napoleonic decision

Major Alexander P. de SeverskyTaken from Davis, 1986


Remember:

Average Porosity

Rig Cost

Recovery Factor

Gross Rock Volume

Fiscal Terms

Net:Gross

Saturation

Continuity

Oil Price

Facilities


Take more core



Buy more seismic

Fire lawyers

More gamma logs


Survey Analogues

Hedge with futures


Action

Expected Value


The industry has been over-focussed on: 1. Developing and using ever more efficient tools for trying to

come up with precise “single number” estimates2. Modelling approaches based on simply “wrapping” a

probabilistic framework around our (very efficient!) classical deterministic tools

3. Modelling information (uncertainty) in the absence of its impact on decisions

How?


Peter Drucker


On Models

• Once you accept decisions really are made under uncertainty, and can be optimally so, it impacts the whole way you view “technical” work (geological, engineering, economic, commercial, legal)

“All models are wrong, some models are useful”

Box

“I would rather be vaguely right than precisely wrong”

Keynes


How?

Model “Richness”HiLo

Lo

Hi

IntegratedUncertainty Evaluation

(No. of“runs” orScenarios)

Too expensiveMissing the

point (ranking)High P(wrong)

Two Components

1. Modelling “philosophy”

2. Type of ModelsClassicalmodels

(precisely wrong)

Vaguely Right (opportunity to create

value)

Trade-off some “domain” rigor for ability to perform integrated uncertainty evaluation


Decision, or feed to portfolio

Monte Carlo Simulation

Sens

itivi

ty A

naly

sis

G&GModel

ProductionModel

DrillingModel

ProcessingFacility Model

ExportModel

Scheduling of Decisions & Implementation

Economics

Taxes &Royalties

Costs

Prices

Scenario and Decision Analysis

Classical Modeling

Uncertainty Estimates, Calibration, Surrogates

Prob

Stochastic Integrated Asset Modelling System (Oil&Gas). Insightful?


The role of “Classical” Modeling

• Uncertainty Estimates of Input Parameters– Simple sensitivity studies

• Calibration of simplified models • Generation of high-fidelity simple surrogates

– Experimental Design and Response Surface Modeling

Classical Modeling

Uncertainty Estimates, Calibration, Surrogates

Holistic Evaluation System


End

Middle

Start

ClassicalApproach

SIAM-basedApproach

Study

Model Scope

Mod

el D

etai

l

BroadNarrow

Sim

plifi

edD

etai

led

Application of Holistic Modelling System


Outline

• Uncertainty, Decisions & Modeling– Uncertainty & Business

Performance– Uncertainty & Decision-

making– Need for integrated models

• Uncertainty, Judgments & People– The Nature of Uncertainty– Uncertainty & Value

Maximization


Probability: The Language of Uncertainty

• Classical (Theoretical)Number of outcomes representing the occurrence of an event

Total number of possible outcomes– e.g. 30 red balls and 70 green balls in a bag. P(Red) = 30%

• Relative Frequency – Proportion of times an event occurs in the long run– Estimated from sample data ASSUMING identical events

e.g. 15 out of 20 wells drilled were dry holes. P(Dry) = 75%– More accurate with greater sample size. May not apply to future.

• Subjective– Personal degree of belief of the likelihood of a future event occurring

(or of the unknown outcome of a past event)– May be based on some past similar / analogous occurrences

“All business proceeds on beliefs, or judgments of probabilities, and not on certainties".

Charles W. Eliot


The nature of uncertainty

Throw a die and hide top face. What is the probability of a 3?

What is the probability of a 3 now?

Now you getinformation.

Uncertainty is a function of what you know. There is no “right” uncertainty (or PDF)!

1/6

Has the top face changed? NoHas the probability of a 3 changed? Yes!

1/3


0.333

1 2 3 5 64Outcome

Prob

.

1 2 3 5 64Outcome

Prob

.

0.333

Uncertainty is in OUR heads – it’s personal, a function of our state of knowledge

Different people can, legitimately, hold different views about the uncertainty of an unknown quantity

Person A Person B

So we should not talk about THE probability of some outcome/statement, but about MY (or OUR) probability


Uncertainty is in OUR heads – it’s a function of our state of knowledge

Probability is not an inherent “parameter” of the “system”!

• The systems that we deal with are essentially deterministic, its just our knowledge of them that is probabilistic

• The Die:- when the die is tossed, there will be one face that comes up

- in theory, if we knew (precisely) its initial conditions and we could model (precisely) all of the processes involved with tossing it, we could predict how it would land – but that is, practically, impossible

• And so with Project reserves,cost, schedule etc:- its our knowledge that is probabilistic

- for events that have occurred, we could collect information to reduce uncertainty - complete information would resolve it


Probability is subjective (personal) and depends upon your information

Person or

CompanyA

Info

SharedInfo

A’s PDF B’s PDF

Person or

CompanyB

Info


Uncertainty vs Variability

Variability of all sand-body widths

Width

Uncertainty in individual sand-body width??

Width

Sand 1

?? Sand 2

A distribution that describes the variability

of a natural phenomenon is not usually appropriate to describe the uncertainty

of a single occurrence


Uncertainty vs Risk

Uncertainty Risk

• A Risk (noun!) is one possible consequence of uncertainty. It has a negative connotation, which is “personal” to the D-M

– an event that, if it occurs, has a negative impact on DM objectives– it is specified by defining the event and assessing its probability,


Risk isn’t the only possible aspect, or consequence, of uncertainty

Consequences of Uncertainty

Risk Possibility of loss or injury

A dangerous element or factor

The degree of probability of loss

Opportunity

Possibility of exceeding expectations

Upside potential

A wonderful element or factor

• Risk is one outcome of uncertainty - but so is Opportunity!– often over-looked - is a source of value creation


Uncertainty, decisions & people

• Bias in central value (mean): – eg “rose-tinted glasses”

• Bias in width of distribution:– eg assessing the range of uncertainty to be much less

than it really is with respect to your true state of knowledge

With respect to uncertainty, the main enemy of good decision making is bias, not the uncertainty being too

great


Common biases and traps in judgment and probabilistic assessment

• Illusion of control

• The confirmation trap

• Overconfidence

• Availability and Vividness

• Anchoring

• The Law of Small Numbers

• Intuition and Repetition

• Hindsight Foresight and the “curse of knowledge”

• Knowing when enough is enough

Human beings are not endowed with rational probabilistic thinking and optimal

behaviour under uncertainty.

Bias & error => poor decisions and judgements

=> undesirable outcomes


Overconfidence Results: Large O&G Industry Sample

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 1 2 3 4 5 6 7 8 9 10

Questions Correct /10

Prop

ortio

n of

Par

ticip

ants Expected

Observed


Does it Matter? Economic impact of over-confidence in O&G project

Estimated PDF (Biased)

12500 15000 17500 20000 22500Area

• Field investment decision where the development plan depended on the reserve estimate.

• Investigate the impact on value (NPV) of overconfidence in reserve parameters, eg

P10 of biased (overconfident) PDF

“True”, unbiased PDF

= P20 of unbiased PDF if the estimatedone was 20% over confident (20OC)Welsh, Begg & Bratvold (2007) SPE 110765


-100

0

100

200

300

400

0% 5% 10% 15% 20% 25% 30%

E(N

PV),

$Mill

ion

Overconfidence

True NPV

EVusing EVs of inputs

Economic Impact of Overconfidence in O&G project

Assessed (Overconfident) Value


Value of Flexibility to Exploit Upside Potential:Extra well slots to manage Volume uncertainty

NPV

$600$200$0EMV = $260

100 kbd PlatformHigh = 30%

Med = 40%Low = 30%

$300$250$100EMV = $220

High = 30%

Med = 40%Low = 30%

60 kbd Platform

Expand

No action

Expand

No action

Expand

No action

Flexible Platform

High = 30%

Med = 40%

Low = 30%

EMV = $303

$180

$270

$580$270

-$20

$70


Value of Flexibility to Exploit Upside Potential:Extra space to manage resource uncertainty

NPV

$600$200$0EMV = $260

100 kbd MillHigh = 30%

Med = 40%Low = 30%

$300$250$100EMV = $220

High = 30%

Med = 40%Low = 30%

60 kbd Mill

Expand

No action

Expand

No action

Expand

No action

FlexibleDevelopment

High = 30%

Med = 40%

Low = 30%

EMV = $303

$180

$270

$580$270

-$20

$70


Summary

• The main purpose of technical work is assessment of uncertainty to aid decision-making - the focus should be ranking, not “best prediction”

• Modeling approach should be decision-driven- holistic models that focus on uncertainty & the complexities

of the system – not the most accurate physics/geology/…• Uncertainty is a function of what we know about a

situation – its in our heads, not a “system” parameter- there is no single, “right” probability for an uncertain event- variability is not the same thing as uncertainty

• Accurate, unbiassed, uncertainty assessment is required to assess the value of decision alternatives


Acknowledgements & Further Info

• People– Prof. Reidar Bratvold, University of Stavanger, Norway– Dr. Matthew Welsh, University of Adelaide– Prof. Michael Lee, University of California, Irvine

• Papers (downloadable from www.onepetro.org)– SPE 71414: “Improving Investment Decisions with a Stochastic

Integrated Asset Model”

– SPE 77509: “Would You Know a Good Decision if You Saw One?”

– SPE 77586: “The value of Flexibility in Managing Uncertainty in Oil & Gas Investments”

– SPE 96423 “Cognitive Biases in the Petroleum Industry: Impact and Remediation”


O&G Evidence of Bias (IPA data)

0 50 100 150 200 250 300

Basis for development sanction

All 1000+projectsin the study

No projects

0% 50% 100% 150% 200% 250% 300%

If the Forecastedproduction is the “Base Case”, we should have approximately as many projects producing more than expected as less than expected !!


Gambling (probability = repeated outcomes) vs. “Real World” (probability = degree of belief)

Uncertainty Quantification

KnownDistribution

Type

UnknownDistribution

Type

KnownParameters

UnknownParameters

1. Identify Possible Outcomes 2. Assign Probabilities to Outcomes

Games of Chance,Geostatistics?

Oil & Gas:Subjective

AllIdentified

Some missed or unknowable

Some denied or ignored


Understanding Risk attitudes

Play game 1

YesOutcome

Tails

Heads

50%

50%

No0

(1)

3

E=1

(10)

30

E=10

(100)

300

E=100

(1,000)

3,000

E=1,000

(10,000)

30,000

E=10,000

(100,000)

300,000

E=100,000

(1,000,000)

3,000,000

E=1,000,000


Understanding Risk attitudes

Play game 2

Outcome

Tails

Heads

Yes

No

50%

50%

0

3

1

E=1.5

30

10

E=15

300

100

E=150

3,000

1,000

E=1,500

30,000

10,000

E=15,000

300,000

100,000

E=150,000

3,000,000

1,000,000

E=1,500,000


Risk Attitudes – Formal Definitions

• Risk-Averse– Prefer a sure option (P=1) whose

value is less than or equal to the EV of an uncertain option

– we put extra “value” on certainty

• Risk-Neutral– No preference between a sure

option (P=1) and an uncertain option whose EV is the same (as the sure value)

EV

PDF Uncertain Option

Sure option

EV

PDF

Sure- prefer this

Uncertain

• Risk-Seeking– Prefer an uncertain option whose

EV is less than or equal to a sure option (P=1)

– we put extra “value” on uncertaintyEV

PDF Uncertain - prefer this

Sure


Impact of Risk Attitudes

• In both cases (risk-seeking and risk-aversion) the loweractual value option is preferred (chosen)!

• Therefore both Risk-Aversion and Risk-Seeking attitudes lead to lower total long-run (multiple-decision) outcomes!– this is consistent with earlier statement that EV maximizes $ value –

any other criterion loses $ value

• E.g. assuming the EV lies between the P10 and P90 then using either of the following decision criteria– invest if P10>0 (very risk-averse), or

– invest if P90>0 (very risk-seeking)

are both value-losing compared to using EV– invest if EV>0 (risk-neutral)


Knowing When Enough is Enough

• In our desire to reduce uncertainty, we often ask for too much information

• We believe –mistakenly – that more information will increase accuracy

• More information helps only to the extent we can use it intelligently

Confidence does increase

Accuracy does not increase

5 10 20 40

10%

20%

30%

40%

Items of information available

% correctpredictions


Heuristics, Biases, Uncertainty & Decisions

• Heuristics– simple rules of thumb and mental shortcuts

• Biases– systematic errors that can result from the use of heuristics

• With respect to uncertainty, the main enemy of good decision making is bias, not the uncertainty being too great

– bias in central value (mean): eg “rose tinted glasses”

– bias in width of distribution: eg assessing the range of uncertainty to be much less than it really is with respect to your true state of knowledge

• Our “mental wiring” is just not good when it comes to uncertainty

– Intuition and “gut feel” often significantly wrong



Bias & error => poor decisions and judgements

=> undesirable outcomes

From http://www.freegifthome.co.uk/blog/best-optical-illusions/- may not be original source


Estimate the gray %

Using a scale of 0% (black) to 100% (white) estimate the % gray of squares A and B


*Shepard’s Table


Perceptual Limitations – visual illusions are a metaphor for cognitive illusions

• Awareness of the illusion, by itself, does not produce a more accurate perception.

• Illusions, therefore, can be extremely difficult to overcome.



Bias & error => poor decisions or judgements

=> More frequent undesirable outcomes


GeologyReservoir

DrillingFacilitiesGeophysics

Asset/Project Evaluation Portfolio Mgmt.

Psychology of Judgments & Decision-making

Decision-Making Under Uncertainty

Actual Performance

Cognitive Science


Main Research Themes

1. Psychological (cognitive science) aspects of judgment and decision-making– Elicitation of un-biased (accurate) expert knowledge for

input to decisions – Matching appropriate decision tools and processes to

decision types– Overcoming barriers–to-adoption

2. Development of models and processes for improved asset/project/portfolio economic valuation and decision-making under uncertainty – Incorporating “real option” thinking – valuing learning (incl

information gathering) and flexibility. – Focus on accurate, rather than ‘precise’, holistic (systems)

models


CIBP Publications: Phase 3

1Welsh, M. & Navarro, D. (in press). Seeing is Believing: Priors, Trust and Base Rate Neglect. Organizational Behavior and Human Decision Processes. Accepted February 7th 2012.

2Willigers, B.J.A., Begg, S., & Bratvold, R.B. (2011). Valuation of Swing Contracts by Least-Squares Monte Carlo Simulation” SPE Economics & Management, Vol 3, No 4, pp 215-225.

1&2Welsh, M. & Begg, S. (in press). Personal ruin versus corporate profit: why individual risk attitudes lessen economic outcomes. APPEA Journal. Accepted 10th January 2012.

1Sykes, M., Welsh, M. & Begg, S. (2011). SPE 146230 - Don't Drop the Anchor: Recognizing and Mitigating Human Factors When Making Assessment Judgments Under Uncertainty. Proceedings of the 2010 Society of Petroleum Engineers Annual Technical Conference and Exhibition. Austin, TX: SPE.

1Bruza, B, Welsh, M, Navarro, D. & Begg, S. (2011). Does anchoring cause overconfidence only in experts? In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1947-1952). Austin, TX: Cognitive Science Society.

1Welsh, M., Delfabbro, P., Burns, N. & Begg, S. (2011). Individual differences in anchoring: numerical ability, education and experience. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 3193-3198). Austin, TX: Cognitive Science Society.

1Welsh, M, Navarro, D. & Begg, S. (2011). Number preference, precision and implicit confidence. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1521-1526). Austin, TX: Cognitive Science Society.

1Welsh, M., Alhakim, A., Ball, F., Dunstan, J. & Begg, S. (2011). Do personality traits affect decision making ability: can MBTI type predict biases? APPEA Journal, 51(1), pp 359-368.



2Willigers, B.J.A., Begg, S., & Bratvold, R.B. (2010). Valuation of Swing Contracts by Least-Squares Monte Carlo Simulation” Paper #133044 in Proc SPE Asia Pacific Oil and Gas Conference, Brisbane.

1Bratvold, R.B. and Begg, S.H. (2010) Making Good Decisions. Richardson, Texas, USA: Society of Petroleum Engineers.

2Bratvold, R.B., Begg, S.H. & Rasheva, S. (2010) A New Approach to Uncertainty Quantification for Decision Making, Paper #130157, in Proc. SPE Hydrocarbon Economics and Evaluation Symposium, Dallas, TX, Mar 8-9

Welsh, M. (2010). Of parrots and parsimony: reconsidering Morgan's canon. Proceedings of the Cognitive Science Conference.

1Bruza, B., Welsh, M., Navarro, D. & Begg, S. (2010). Effect of presentation order and question format on subjecive probability judgments. Proceedings of the Cognitive Science Conference.

1Mackie, S., Begg, S., Smith, C. & Welsh, M. (2010). Human Decision-Making in the Oil and Gas Industry. Proceedings of the Society of Petroleum Engineers Asia Pacific Oil and Gas Conference.

1Welsh, M. & Begg, S. (2010). Don't let it weigh you down: how to benefit from anchoring. Proceedings of the 2010 Society of Petroleum Engineers Annual Technical Conference and Exhibition.

1Welsh, M., Rees, N., Ringwood, H. & Begg, S. (2010). The Planning Fallacy in Oil and Gas Decision Making. APPEA Journal, 50 (1), pp 389-401.



1Mackie, S., Welsh, M. & Lee, M. (2006). An oil and gas decision-making taxonomy. Proceedings of the 2006 Asia Pacific Oil and Gas Conference of the Society of Petroleum Engineers. Richardson, TX: SPE.

1Welsh, M., Begg, S. & Bratvold, R. (2006). Correcting common errors in probabilistic evaluations: efficacy of debiasing. Proceedings of the 82nd Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.

2Al-Harthy, M., Khurana, A., Begg, S., Bratvold, R., (2006) Sequential and Systems Approaches for Evaluating Investment Decisions: Influence of Functional Dependencies and Interactions. APPEA Journal, Vol 46, part 1, pp511-523

2Chapman, T., Nettelbeck, T., Welsh, M. & Mills, V. (2006). Investigating the construct validity associated with microworld research: a comparison of performance under different management structures across expert and non-expert naturalistic decision-making groups. Australian Journal of Psychology, 58(1), pp. 40-47. Drop by to look at the

2Yao, Y., Begg, S.H., Bratvold, R.B., Behrenbruch, P., van der Hoek, J. (2006) "A Case Study for Comparison of Different Real Option Approaches", Paper # 101031 in Proc. SPE Asia Pacific Oil and Gas Conference and Exhibition Adelaide, Australia, September, 2006.

2Bratvold, R. and Begg, S.H. (2006) "Education for the Real World: Equipping Petroleum Engineers to Manage Uncertainty", Paper #103339, in Proc. SPE ATCE, San Antonio

1Welsh, M., Bratvold, R. & Begg, S. (2005). Cognitive biases in the petroleum industry: impact and remediation. Proceedings of the 81st Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.

2Bratvold,R.B., Laughton,D., Enloe,T., Borison,A., Begg, S.H., (2005) “A Critical Comparison of Real Option Valuation Methods: Assumptions, Applicability, Mechanics, and Recommendations”, Paper #97011, in Proc. SPE ATCE, Dallas



1Welsh, M. & Begg, S. (2009). Repeated judgment elicitation: tapping the wisdom of crowds in individuals. Proceedings of the 85th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers

1Welsh, M., Lee, M & Begg, S. (2009). Repeated judgments in elicitation tasks: efficacy of the MOLE method. In N.A. Taatgen & H. van Rijn (Eds), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp1529-1534). Austin,TX:CognitiveScience Society.

1Bruza, B., Welsh, M. & Navarro, D. (2008). Does Memory Mediate Susceptibility to Cognitive Biases? Implications of the Decision-by-Sampling Theory. In V. Sloutsky, B. Love, & K. McRae (Eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society.

1Heywood-Smith, A., Welsh, M. & Begg, S. (2008). Cognitive Errors in Estimation: Does Anchoring Cause Overconfidence? Proceedings of the 84th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers.

1&2Welsh, M. & Begg, S. (2008). Modeling the Economic Impact of Individual and Corporate Risk Attitude. Proceedings of the 84th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers.

1Welsh, M., Lee, M. & Begg, S. (2008). More-Or-Less Elicitation (MOLE): Testing a heuristic elicitation method. In V. Sloutsky, B. Love, & K. McRae (Eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society.

2Begg, S.H., and Bratvold, R.B. (2008) Systematic Prediction Errors in O&G Project and Portfolio Selection., Paper #116525, in Proc. SPE ATCE, Denver.

2Bratvold, R. and Begg, S.H. (2008) I would rather be Vaguely Right than Precisely Wrong , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1373-1392



2Cunningham, P. and Begg, S.H., (2008) Using Value of Information to Determine Optimal Well Order in a Sequential Drilling Program , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1393-1402

2Smalley, P.C., Begg, S.H., Naylor, M., Johnsen, S. & Godi, A. (2008) Handling Risk and Uncertainty in Petroleum Exploration and Asset Management: An Overview , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1251-1261

1&2Mackie, S., Begg, S., Smith, C. & Welsh, M. (2008). "Real World" Decision-Making in the Upstream Oil and Gas Industry - Prescriptions for Improvement. APPEA Journal, 48(1), pp 329-343.

1&2Welsh, M., Begg, S. & Bratvold, R. (2007). Modeling the economic impact of cognitive biases on oil and gas decisions. Proceedings of the 83rd Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.

1Welsh, M., Begg, S. & Bratvold, R. (2007). Efficacy of bias awareness in debiasing oil and gas judgments. Proceedings of the 29th Annual Conference of the Cognitive Science Society.

1Welsh, M., & Navarro, D. (2007). Seeing is believing: priors, trust and base rate neglect. Proceedings of the 29th Annual Conference of the Cognitive Science Society.

1&2Mackie, S., Begg, S., Smith, C. & Welsh, M. (2007). Decision type - a key to realizing the potential of decision-making under uncertainty. APPEA Journal, 22(1), pp. 307-317.

2Al-Harthy, M., Begg, S., Bratvold, R., (2007) “Copulas: A New Technique to Model Dependence in Petroleum Decision Making”. Journal of Petroleum Science and Engineering, 57, pp195-208

2Begg, S.H., and Smit, N. (2007) “Sensitivity of Project Economics to Uncertainty in Type and Parameters of Oil Price Models“, Paper #110812, in Proc. SPE ATCE, Anaheim

1Elliot, T., Welsh, M., Nettelbeck, T. & Mills, V. (2007). Investigating Naturalistic Decision Making in a simulated micro-world: What questions should we ask? Behavior Research Methods, 39(4), pp 901-910.



Lee, M., Pincombe, B. & Welsh, M. (2005). An empirical evaluation of models of text document similarity. Proceedings of the 27th Annual Conference of the Cognitive Science Society. Mahwah, NJ: Erlbaum

1Welsh, M., Begg, S., Bratvold, R. & Lee, M. (2004). Problems with the elicitation of uncertainty. Proceedings of the 80th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.

2Begg, S.H. and Bratvold, R., (2004) “The value of flexibility”, Southern China Oil & Gas Journal, vol17, no.4, p59.

2Begg, S.H., Bratvold, R., Campbell, J.C, (2004) “Abandonment Decisions and The Value of Flexibility “, Paper #91131, in Proc. SPE ATCE, Houston

2Laughton, D., Bratvold, R., Begg, S.H., Campbell, J.C. (2004) “Development as the continuation of appraisal by other means “, Paper #90155, in Proc. SPE ATCE, Houston

2Lee, M., O'Connor, T. & Welsh, M. (2004). Decision making on the full information secretary problem. Proceedings of the 26th Annual Conference of the Cognitive Science Society. Mahwah, NJ: Erlbaum.

1&2Begg, S.H., Bratvold, R., Campbell, J.C. (2003) “Shrinks or Quants: Who will improve decision-making? “, Paper #84238, in Proc. SPE ATCE, Denver

2Bratvold, R., Begg, S.H., Campbell, J.C. (2003) “Even Optimists should optimize“, Paper #84329, in Proc. SPE ATCE, Denver

2Begg, S.H., Bratvold, R., Campbell, J.C, (2003) “Decision-making Under Uncertainty”, In Proc. 7th

International Symposium on Reservoir Simulation, Baden-Baden 2Campbell, J.C, Begg, S.H., Bratvold, R.B. (2003) “Portfolio Optimization: Living up to Expectations?“,

Paper #82005, in Proc. SPE Hydrocarbon Economics and Evaluation Symposium, Dallas


Outline

• Introductions, Questionnaire• Overview of Performance of O&G

Industry Capital Investment Decisions • Underlying Concepts• Major Psychological Factors• Limits of Intuition in Complex & Uncertain

Situations• What can we do… RISC Process


Uncertainty Assessment Exercise

Question 1

Question 2

Question 3

etc

Lower Limit(P10)

UpperLimit(P90)

80%Chance

10% 10%

172 cm

E.g. What is my height in cm?

• Your goal is to set ranges such that they are- Narrow enough not to contain the actual value more than 8 out 10

times (80%) on average, or

- Wide enough to contain the actual value 8 out of 10 times


Anchoring QuestionLarge Industry Sample• Question given to two groups. One group had a High

anchor, other group had Low anchor

• High“Were world proved oil reserves in 2003 greater or less than 1721Billion Barrels?” Yes [ ] No [ ]

• Low“Were world proved oil reserves in 2003 greater or less than 574Billion Barrels?” Yes [ ] No [ ]

• Both versions then asked“What is your best estimate of the world proved oil reserves in 2003?”

( )


Anchoring ResultsLarge Industry Sample

19321722

574 682

0

500

1000

1500

2000

2500

3000

3500

Low High

Anchor Group

Mea

n Es

timat

ed W

orld

Pro

ved

Res

erve

s 20

03 +

/- 1s

d AnchorEstimate

Common approach in E&P project evaluation:

“Let’s start with a base case and then build some scenarios around it.”


Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes

• Overconfidence, Optimism and Superiority Biases

• Anchoring

• Unpacking & the Planning Fallacy

• Availability, Recency and Vividness

• Hindsight Bias


REMOVE THIS SLIDE• I am not convinced about giving these definitions (next 3

slides) up front – liable to lead to unfruitful discussion to distinguish them before they have really learnt what they are

• I also really struggled to make / modify the illustrative “quotes” to keep them pithy and descriptive of the bias/factor


RISC view of primary Psychological Factors affecting E&P projects

Overconfidence: “There is an 90% chance the cost will be less than $10MM”; placing higher probabilities on events (or tighter probability distributions) than is warranted by our true state of knowledge. Due to this bias, actual outcomes will lie outside our ranges more frequently than expected.

Optimism: “There is only a 5% chance of a delay of more than 3 months”; assigning lower chances to the attainment of undesirable outcomes (and/or higher chance to desirable outcomes) than objective criteria, experience or logical analysis warrants. Due to this bias, outcomes will be systematically worse than expected.

Positive Illusions (Superiority Bias): “We can do it better than anyone else” causes people to overestimate, relative to others, their positive qualities, skills and abilities and to underestimate their negative qualities - leading them to believe that THEY are less at risk of experiencing a negative event compared to others who are doing the same thing. Also leads to the Illusion of Control - under attributing the role of chance, in decision outcomes



Anchoring: “I wish the time to build XXX was less than 4 months”; an initial piece of information, typically a number (perhaps irrelevant) causes people (often sub-consciously) to “centre” on that information and to not adjust sufficiently far away when considering other possibilities

Planning Fallacy: “If everything goes to plan we’ll be on-line in 2 years” a tendency for people and organizations to rely on best-case scenarios, leading to optimistic estimates of how long it will take to complete complex tasks, even when they have experience of similar tasks over-running (their own or others).

Packing/Unpacking Effect: “lets not make this too complex and go with a high-level breakdown of tasks”

When broad tasks, or causes, are broken down (unpacked) into explicitly-identified sub-components, the estimates of times, costs, %-contribution for each task, and thus the overall total for all tasks, is more accurate. Failure to unpack adequately is thought to be a contributor to the planning fallacy – “out of sight is out of mind”



Availability, Recency & Vividness: “I have an excel spreadsheet handy with cost estimates from our last project .....”

causes individuals to over-weight the most easily accessed, remembered or recent information when assessing the likelihoods of the possible outcomes of future events

Hindsight bias: “I knew that was likely to happen, ...”;the inclination, in retrospect, to assign higher chances to outcomes that have already occurred than were assigned before the event took place –can apply to one’s own, or other peoples, estimates of the chance of the outcome occurring




• Anchoring



• Hindsight Bias


Refresher: Interpreting PDFs & Percentiles

0.10

P10 P90

80% ofarea

10% ofarea

10% ofarea

The area under the PDF between any two points is theprobability of X lying between those two points


Uncertainty Assessment Exercise

Question 1

Question 2

Question 3

Question 4

Question 5

Question 6

Question 7

Question 8

Question 9

Question 10

Lower Limit(P10)

UpperLimit(P90)

Actual80%Chance

10% 10%


Probability Estimation Exercise

• Excel - Interactive


*Assessing Uncertainty

• Some conclusions from Capen study (3000+ sample):– People who are uncertain about answers to a question have

almost no idea of the degree of their uncertainty. They cannot differentiate between a 30- and a 98-percent probability interval

– The more people know about a subject, the more likely they are to construct a large probability interval (one that has a high chance of catching the truth), regardless of what kind of interval they have been asked to use. The converse seems to hold as well; the less known, the smaller the chance that the interval will surround the truth

– People tend to be a lot prouder of their answers than they should be

– Even when people have been told that probability ranges tend to be too small, they cannot bring themselves to get their ranges wide enough


Time

Min

Max

Discovery

Conceptselect

Detailed design

Practical completion

Procurement

Concept Construction CommissionDesign

Perceived uncertainty

Actual uncertainty

* Experts & Over-confidence


We’re not alone: Experts & Over-confidence

• Heavier-than-air flying machines are impossible

– Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, spoken in 1895

• A severe depression like that of 1920-21 is outside the range of probability

– Harvard Economic Society, Weekly Letter, November 16, 1929



• That idea is so damned nonsensical and impossible that I'm willing to stand on the bridge of a battleship while that nitwit tries to hit it from the air

– Newton Baker, U.S. secretary of war in 1921, reacting to the claim of Billy Mitchell (later Brigadier General Mitchell) that airplanes could sink battleships by dropping bombs on them



• They couldn't hit an elephant at this dist…

– General John B. Sedgwick, Union Army Civil War officer's last words, uttered during the Battle of Spotsylvania, 1864

• I think there is a world market for about five computers

– Thomas J. Watson, chairman of IBM, 1943


We’re not alone: US DoE Price Forecasts

Trends Predicted Beginning From the Actual Price of Year Listed

after U.S. Department of Energy, 1998

120

100

80

60

40

20

01975 1980 1985 1990 1995 2000 2005

Year

1982

1981

1984

1985

1986 1987

1991

1995Actual

Dol

lars

per

Bar

rel

1983


We’re not alone: Yr 2000 Price Forecasts

U.S. Department of Energy, 1998

35

30

25

20

15

102000 2005 2010 2015 2020

IEA

DOE High

Mobil

DRI

DOE Base

Nat. Res. Canada

Nat. West Sec.

DOE Low

Pet. Econ. Ltd.

$/B

BL

(199

6 D

olla

rs)

Year

Widely Divergent ForecastsMake Planning Difficult


Does it matter? Overconfidence in the Stock Market

• Based on a study of monthlypositions and trading records of 88000 investors over a 10 year period and $2 million commonstock trades (Barber & Odean 2000).

• In contrast to a buy-and-hold strategy, the average investor traded 75% of their investment in any given year.

• The investors traded frequently because they thought they could beat the market


Overconfidence in the Stock Market: You Are What You Trade

• The average investor earned a return of 16.4% during the booming market.

• The overall market return for this same period was 17.9%.

• The 20% of accounts (more than 12000) that had the highest turnover rates earned a return of just 11.4%.

• Overconfident investors under diversify.


Overconfidence in the Stock Market Boys will be Boys

• The data (Barber & Odean 2001) show that women performed better than men

– Not because they were better at picking stocks but because …

• Men tend to be more overconfident than women.– Men trade 45 percent more actively than women.

– For single men and women, the difference is 67%.


Calibration

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Estimated Accuracy (Confidence)

Act

ualA

ccur

acy

OverConfidence

UnderConfidence

PerfectCalibration

Typical O&Gresult at 80% confidence

• Calibration: the degree to which estimated accuracy (Confidence) matches actual accuracy


Expert Calibration: Physicians

Data: Physicians, after completing history and physical examination, estimated the probability that patients had pneumonia (Source: Christensen-Szalanski & Bushyhead, 1981)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Subjective Probability of Pneumonia

% R

adio

grap

hica

lly a

ssig

ned

pneu

mon

ia


What Can We Do?Expert Calibration: US Weather Forecasters

Source: Russo and Schoemaker

• Why are physicians lousy and weather forecasters great?- A significant reason is because they get frequent, immediate, and

accurate feedback

161

146

282575

589

257

172

203147

159 82

38 9

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Forecast Probability of Rain

Act

ual F

ract

ion


In the resources industries pressures towards overconfidence are rife

• Experts may feel pressure (motivational or cognitive) to demonstrate their expertise relative to peers or competitors and therefore place overconfident bounds on uncertain quantities

– Narrower bounds are used to imply “I(we) know more than you”

• Managers may create a climate that discourages a true assessment of uncertainty:

– “you are paid to know, not to not know …”

– “you are just covering your …”

– “why can’t you just tell me the answer”

– “I need a single number to make a decision”


Optimism

• A bias towards – assigning higher chances to the attainment of desirable outcomes– assigning lower chances to the attainment of undesirable outcomes

than objective criteria, experience or logical analysis warrants.

• Optimism can be the result of– Dispositional: a personality trait– Situational: e.g. a motivational bias, caused by an incentive to be

optimistic rather than realistic• This bias raises the odds that the projects chosen for

investment will be those with the most optimistic forecasts –and hence the highest probability of disappointment

– outcomes will be systematically worse than expected (worse being context dependent – ie higher costs, lower production)


Overconfidence v Optimism: Impact on Assessed Probabilities, Pictorially

“True” Uncertainty

Over-Confidence- assigning too-narrow a PDF

compared to your true state of knowledge

- driven by mis-assessing our level of knowledge

Optimism- assigning higher chance to

more desirable outcomes (eg“lower” valued outcomes, such as costs, are better)

- driven by our personality or by preferences for some of the outcomes


Overconfidence & Optimism!

“True” Uncertainty

- in this case “higher” outcomes are more desirable (egproduction rates)

Over Confident and Optimistic


Optimism and Organizational Pressure: Motivational perspective• Every company has limited investment funds and time to

devote to new projects.– Competition for this time and money is intense, as individuals and units

present their own proposals as being the most attractive for investment.

• The selection process often– favours the most communicative and articulate – not necessarily the most

knowledgeable

– discourages “realism” (unbiasedness), encourages exaggeration

– causes bearers of bad news, or people who point out problems, to be labelled as “not a team player” - we do “shoot the messenger”!

Big incentives to accentuate the positive in project forecasts.

Ability of the organization to think critically is under-minded

Optimistic individual views are self-reinforcing and unrealistic views of the future are validated by the group


Using outcomes of multiple estimates to check for unbiasedness

P20 P60 P100P0 P40 P80

% outcomes lying within estimated range

20%Unbiased

!! !!P20 P60 P100P0 P40 P80

20%Biased% outcomes

lying within estimated range


Questions

how good a decision-maker are you?

• Compared to your peers, on a scale of 1 to 10, where- 1 is poor- 5 is average - 10 is very goodhow good a driver are you?

how good is your intuition?how good a probability estimator are you?


Positive Illusions (Superiority Bias)

• People/organizations overestimate their positive qualities, skills and abilities (and underestimate their negative qualities) relative to other people/organizations

• This can lead them to believe that THEY are less at risk of experiencing a negative event (e.g. cost/time overrun) compared to OTHERS in the same situation.


*Lake Wobegon effect:“… and all the children are above average” – Garrison Keillor

• 82% of people say they are in the top 30% of safe drivers;

• 86% of MBA students say they are better looking than their classmates;

• 68% of lawyers in civil cases believe that their side will prevail;

• Doctors consistently overestimate their ability to detect certain diseases;

• 81% of new business owners think their business has at least 70% chance of success, but only 39% think that any business like theirs would be likely to succeed.

Source: Russo & Schoemaker


Lake Wobegon effect:“… and all the children are above average” – Garrison Keillor

• 82% of people say they are in the top 30% of safe drivers;

• 86% of MBA students say they are better looking than their classmates;

• 68% of lawyers in civil cases believe that their side will prevail;

• Doctors consistently overestimate their ability to detect certain diseases;

• 81% of new business owners think their business has at least 70% chance of success, but only 39% think that any business like theirs would be likely to succeed.

Source: Russo & Schoemaker

• When assessing their position in a distribution of peers on almost any positive trait, 90% of people say they are in the top half.


*Illusion of Control

OutcomeDeciding

• the thinking and decision process

Doing• implementation

and other factorsunder your control

Chance• uncontrollable

factors, luck

Skill Presumed CauseSuccess

FailurePresumed Cause


Illusion of Control

OutcomeDeciding

• the thinking and decision process

Doing• implementation

and other factorsunder your control

Chance• uncontrollable

factors, luck

Skill Presumed CauseSuccess

FailurePresumed Cause

• Illusion of control frequently causes people to repeat actions that in the past were followed by success.

• This is true even if there’s no reason to believe the actions did anything to cause the success.

• Only by realistically assessing the role of chance in successes can you learn which of your actions you should repeat and which could be improved.


Illusion of Control:

• People often (knowingly and unknowingly) take credit for positive outcomes and attribute negative outcomes to external factors, no matter what their true cause.

• Study of letters to shareholders:– Executives tend to attribute favourable outcomes to factors

under their control, and

– Unfavourable outcomes were more likely to be attributed to uncontrollable external events such as weather or inflation.

“Victory has a thousand fathers; defeat is an orphan.”

-the Duke of Wellington


Thinking about “Luck”

• By definition, luck (or unluck) is something over which you do not have control, therefore:

– People are not inherently lucky or unlucky. – Lucky or unlucky things happen to people.

• People cannot “create their own luck” – but they can plan to exploit good luck when it happens,

and minimize the impact of bad luck when that happens– “Plan”: by creating an environment, contingencies or

opportunities to respond to events, rather than living with fate.


Summary of Overconfidence, Optimism and Positive Illusions (Superiority)

Overconfidence: “There is an 90% chance the cost will be less than $10MM”; placing higher probabilities on events (or tighter probability distributions) than is warranted by our true state of knowledge. Due to this bias, actual outcomes will lie outside our ranges more frequently than expected.

Optimism: “There is only a 5% chance of a delay of more than 3 months”; assigning lower chances to the attainment of undesirable outcomes (and/or higher chance to desirable outcomes) than objective criteria, experience or logical analysis warrants. Due to this bias, outcomes will be systematically worse than expected.

Positive Illusions (Superiority Bias): “We can do it better than anyone else” causes people to overestimate, relative to others, their positive qualities, skills and abilities and to underestimate their negative qualities - leading them to believe that THEY are less at risk of experiencing a negative event compared to others who are doing the same thing. Also leads to the Illusion of Control - under attributing the role of chance, in decision outcomes


Effect of Overconfidence, Optimism and Positive Illusions (Superiority)

• Collectively these bias’s affect the estimates and uncertainty ranges that are generated and used as part of Project cost and schedule workflows

• Ultimately the effect is to provides estimates that are too low and uncertainty ranges that are too narrow

• No-one is immune to these psychological issues and unless people are made aware of them (and even then) they will always skew our work

• The culture and incentive systems in organisations (and Project teams) actively drives overconfidence and optimism in the people within


Mitigating Overconfidence, Optimism and Positive Illusions (Superiority)• Be aware of them (not just now, continually remind yourself)• Ask yourself why you might be wrong (as opposed to right) –

actively think of all the reasons why the P10 might be smaller and the P90 larger.

• Get Feedback– Privately record lots of predictions (probability for discrete events; P10-

P90 ranges for continuous variables) of things you will soon get to know the answer to – and record the outcomes as well.

– Enhance your motivation to be well-calibrated by making small bets with friends, colleagues and family on the outcomes

• Don’t limit your assessment of uncertainty to finding data (of the event in question) and fitting a distribution to them

– Unless the data are very numerous and truly repeat outcomes of the same “event” of interest


Mitigating Overconfidence, Optimism and Positive Illusions (Superiority)• Take an “outside view”

– look at the performance of other people/teams/companies, or how you think others would perform if doing your projects

• Managers/Supervisors– Create a climate where honesty about uncertainty is encouraged– Reward for quality of predictive process – not actual outcome– Ask people to justify why their ranges are so narrow (half the P10,

double the P90)




• Anchoring



• Hindsight Bias


Anchoring - Subtle changes in wording of a question can significantly impact responses

Question OutcomeDo you get headaches frequently? How often do you get them?Do you get headaches occasionally? How often do you get them?

2.2 / week

0.7 / week

Question OutcomeHow long was the movie?

How short was the movie?130 min100 min

OutcomeQuestion OutcomeQuestionHow tall was the basketball player? 79 inches

69 inchesHow short was the basketball player?


Anchoring - Subtle changes in wording of a question can significantly impact responses

Question Outcome

Question Outcome

How wide are the channels?

How narrow are the channels?

How big is the fault throw?

How small is the fault throw?


• Participants watched a one-minute film that include a four-second multiple car crash.

• The participants were then divided into 5 groups who were asked:

– “About how fast were the cars going when they smashed into each other?”

– “About how fast were the cars going when they collided with each other?”

– “About how fast were the cars going when they bumped into each other?”

– “About how fast were the cars going when they hit each other?”

– “About how fast were the cars going when they contacted each other?”

Anchoring and Adjustment


Anchoring and Adjustment

Descriptor Mean Speed

Smashed 40.8

Collided 39.3

Bumped 38.1

Hit 34.0

Contacted 31.8

Ref.: Loftus and Palmer

“About how fast were the cars going when they ??? into each other?”


Anchoring Question

• Group A?

• Group B?


Anchoring & Adjustment

• Describes a heuristic commonly used by people when estimating values.

• Use any given number/statistic/fact as a starting point (anchor) that from then on dominates the thinking process.

– adjust away from there to reach estimate

– generally people adjust too little so their estimates cluster near the anchor

• Random anchors can have just as large effects as credible anchors.

– Quattrone et al (1984) asked whether the average temperature in San Francisco was greater or less than 558º and still found people anchoring on this value

• Subtle changes in wording of a question can have significant impact on how people respond.


Anchoring - Implications

• Referendums and opinion polls.• Jurors:

– “Strike that comment”

• Independence of second opinions.• Skilled negotiators often start by setting a suitable

anchor.• Used car salesmen and real estate brokers.• Resource industry managers?• Common approach in project evaluation:

– “Let’s start with a base case and then build some scenarios around it.”


Summary of Anchoring

Anchoring: “I wish the time to build XXX was less than 4 months”; an initial piece of information, typically a number (perhaps irrelevant) causes people (often sub-consciously) to “centre” on that information and to not adjust sufficiently far away when considering other possibilities


Anchoring: Effect and Mitigation

• Effect- This bias can affect the initial estimates (and then uncertainty ranges)

that are generated and used as part of Project cost and schedule workflows

- A bias that is used extensively when negotiating with someone who is aware of the effect

- Difficult to overcome even when aware of the effect- Framing of questions is extremely important to reduce the effect

• Mitigation- Be aware of it – not just now, remind yourself- Deliberately use multiple Anchors- NEED MORE




• Anchoring



• Hindsight Bias


The Planning Fallacy

• The tendency to underestimate completion times of projects because planners construct a single mental scenario, comprised of broad stages, in which most things go according to plan, – DESPITE knowing that similar projects (whether our own! or others)

have gone over time & budget

• The “because” above is the strict/narrow definition of the Planning Fallacy and is thought to be due to– Failing to distinguish between “best guess” and “best case” scenarios– Failure to unpack tasks, especially “other problems”, or ignoring them

completely

The whole workshop could be considered to be about a broader definition of the “planning fallacy” (lower case) and thus a result of all the psychological factors we are considering

– Overconfidence; Optimism (as a personality trait or motivational bias); Illusion of Control and Superiority bias; ……


The Planning Fallacy & Unpacking Effect

• When asked how long a 1-hour lecture takes to prepare, lecturers estimate around 3 hours

• When asked to estimate how long each task involved in preparing the same lecture takes, the sum of these task times is significantly greater than 3 hours


• 1st Unpacking Question (Packed)– “What % of world proved oil reserves are in

the following areas: Saudi Arabia, Iraq, Omanand All Others?”

• 1st Unpacking Question (Unpacked)– “What % of world proved oil reserves are in

the following areas: Saudi Arabia, Iraq, OmanVenezuela, USA, China, Russia, Nigeria andAll Others?”

Unpacking Questions


Unpacking Question 1 Results

0

10

20

30

40

50

60

70

80

SaudiArabia

Iraq Oman All Others

Region

Estim

ated

% o

f Wor

ld P

rove

d Res

erve

s (2

003)

+/-

1SD

True:PackedUnpacked


Unpacking example: estimating time to drill a well

• Four groups were asked to estimate completion times, in hours, for a real-world drilling scenario– 3rd yr ASP Pet. Eng. undergraduates (no decision-making training)– 4th yr ASP Pet. Eng. undergraduates (some decision-making training)– “Conversion” Masters of Pet. Eng. (little Pet. Eng. Knowledge)– Industry petroleum engineers (with average 10 yrs experience)

• Approximately half were given a Packed version of the scenario which consisted of 4 components

– Drilling, Tripping, Rigging and All associated problems

and the rest given an version where “All associated problems” was Unpacked into 6 explicit possibilities

– Mud conditioning; Well-control operation; Fishing operations; Severe weather; Rig repair; Logistics delays


Unpacking example: estimating time to drill a well

‘An oilfield is located offshore Louisiana in about 350 ft of water. The field contains four wells that are currently producing 1,000 bbl of oil and 18 MMSCF of gas per day. Production is declining and it has been decided to drill an infill well in order to improve production. The well is required to be drilled to 10,000 ft drilling overbalance with mud. For this example, we will assume the well can be drilled straight to 10,000 ft without the need for staged casing. The rig is already in place and ready to commence drilling. From other well data, the stratigraphy can be assumed to be reasonably homogeneous and consolidated. The well needs to be ready up to the point where casing could then be run if required.’

Bourgoyne et al (1986)


Unpacking Results: Number of hours of drilling problems

Unpacked

Packed

Estim

ated p

roble

m ho

urs,

mean

and 9

5% C

I

Group


Unpacking Effect

• Out of sight is out of mind– Explicitly stating an option makes it available and

therefore increases its estimated likelihood– Unpacking a general category into specific

subcategories therefore increases the total likelihood assigned to that category despite the two being logically equivalent

• The Planning Fallacy– A specific instance of the unpacking effect– Refers to the tendency of people to underestimate

completion times for complex tasks


Summary of Planning Fallacy & Unpacking Effect

Planning Fallacy: “If everything goes to plan we’ll be on-line in 2 years” a tendency for people and organizations to rely on best-case scenarios, leading to optimistic estimates of how long it will take to complete complex tasks, even when they have experience of similar tasks over-running (their own or others).

Packing/Unpacking Effect: “lets not make this too complex and go with a high-level breakdown of tasks”

When broad tasks, or causes, are broken down (unpacked) into explicitly-identified sub-components, the estimates of times, costs, %-contribution for each task, and thus the overall total for all tasks, is more accurate.


Mitigating Planning Fallacy & Unpacking Effect

• If you insist on using single-point estimates – make a “best guess” estimate rather than a “best case” estimate

• Unpack general categories (such as “all other problems”) by just listing their sub-components

– this will improve the estimate of the general category, even if you do not estimate each of the subcomponents

• Take an “outside view” – look at the performance of other people/teams/companies, or how

you think others would perform if doing your projects




• Anchoring



• Hindsight Bias


*Availability and Vividness

• Are there more words in English that have the letter ‘k’ as their first letter (e.g., kill) or as their third letter (e.g., ink)?

a) first letter

b) third letter

• Experimental results (Tversky and Kahneman):– 2/3 of people asked thought that words with the letter k

in the first position were more probable

• Reality:– There are approximately twice as many words with k in

the third position as there are words that begin with a k.


*Availability and Vividness

• Which of the following is more likely to kill someone in the USA?a) Shark Attack?b) Falling airplane parts?– Falling airplane parts 30 times more likely than shark

attack in the US (cited in Plous, 1993)

• Which of the following caused more deaths in Australia in 2003a) Renal Failure?b) Car and other transport Accidents?– Renal failure - 15000 compared to 2100 for transport

accidents (Australian Bureau Statistics, 2003)


Availability and Vividness Bias – Seeing What We Believe

• The tendency people have to base estimates of frequencies (probabilities) on the most readily available, recent and vividinformation they can remember

– how many events of a particular type are availableto memory

– more available events are judged more likely

• Memory is limited to 7 “chunks”


Bias from the Availability Heuristic & Saliency Heuristic

• This heuristic leads to bias where it is easier to recall events for reasons other than their frequency, e.g. news coverage

– Shark attacks provoke a media feeding frenzy making them easier to recall

– Car accidents often make the news over long weekends with multiple deaths whereas renal failure only ever affects one person at a time and gets no coverage

• Another contaminating factor in our ability to use the availability heuristic accurately is the salience of events, e.g. how relevant they are to us personally

– People whose houses have burnt down inflate the likelihood of house fires when asked

– Knowing someone who has died of a particular cancer tends to cause inflation of estimates of its occurrence


Bias from the Availability Heuristic & Saliency

• Tversky and Kahneman (1974)– “decision makers assess the frequency of a class or

the probability of an event by the ease with which instances or occurrences can be brought to mind.”

• Managers conducting performance appraisals:– Working from memory, vivid instances of an

employee’s behaviour (either positive or negative) will be most easily recalled from memory and will appear more numerous than more commonplace instances.

– Managers give more weight to performance during the three months prior to the evaluation than to the previous nine months of the evaluation period.


Bias from the Availability Heuristic & Saliency

• Engineers who are evaluating durations for potential project tasks.

• Commonly done by comparing the tasks with tasks that have been done previously and where, as a consequence, the durations are known.

• Potential biases induced with this strategy are– previous personal experience with particular risks can

be overweighted

– no comparison is made with other risks that have not been experienced but that will potentially effect the tasks


Summary of Availability, Recency and Vividness

Availability, Recency & Vividness: “I have an excel spreadsheet handy with cost estimates from our last project .....”

causes individuals to over-weight the most easily accessed, remembered or recent information when assessing the likelihoods of the possible outcomes of future events

• Difficult to overcome due to the amount of information that can be contained in the memory

• Highlights that memory can not be relied on when making estimates

• When recalling from memory, write down a list and try to think of the most common occurrences (not the most vivid or recent)

• Search for the most pertinent data, not the most available




• Anchoring



• Hindsight Bias


Hindsight Bias:Remembered Probabilities of Once-Future Things

• Prior to President Nixon’s trip to China and Russia in 1972, students were asked to consider 15 possible outcomes:– “the US will establish a permanent diplomatic mission in

Peking, but not grant diplomatic recognition,”

– “Nixon will meet Mao Tse-tung at least once,”

– “Nixon will meet Soviet demonstrators,”

– …

• The students assigned probabilities to each outcome.

Ref: Fischhoff and Beyth (1975)


*Hindsight Bias:The “We Knew it All Along” Phenomenon

• After the trip the students were asked in hindsight to assess the likelihood of these various outcomes and also asked to remember or reconstruct their original probabilities.

• Two weeks between pre- and post-trip– 67% thought their original estimates were closer to the truth

then they really were.

• Four to eight months between pre- and post– 84% thought they had predicted the outcome.


Hindsight Bias:The “We Knew it All Along” Phenomenon

• After the trip the students were asked in hindsight to assess the likelihood of these various outcomes and also asked to remember or reconstruct their original probabilities.

• Two weeks between pre- and post-trip– 67% thought their original estimates were closer to the truth

then they really were.

• Four to eight months between pre- and post– 84% thought they had predicted the outcome.

“people even misremember their own predictions so as to exaggerate in

hindsight what they knew in foresight”Fischhoff and Beyth

How will this bias impact our ability to, in hindsight, judge the quality of our

predictons?


Summary Hindsight Bias

• Makes learning from previous estimates much more difficult

• Highlights that memory can not be relied on when doing “lessons learnt” workshops

• Records of previous predictions/estimates, and reasons for them, should be made at the time of the prediction/estimate and checked against actual performance to help with calibrating the mind

Hindsight bias: “I knew that was likely to happen, ...”;the inclination, in retrospect, to assign higher chances to outcomes that have already occurred than were assigned before the event took place –can apply to one’s own, or other peoples, estimates of the chance of the outcome occurring


Outline

• Introductions, Questionnaire• Overview of Performance of O&G Industry

Capital Investment Decisions • Underlying Concepts• Major Psychological Factors• Limits of Intuition in Complex & Uncertain

Situations– Uncertainty Propagation– Updating Estimates with New Information– Complexity

• What can we do… RISC Process


Estimating Odds

• Gilbert Video (2005G) Pt 1 – Odds

00.00-8:19


Judging likelihoods of events

• Linda is a 31 years old, single, outspoken and very bright. She majored in philosophy. As a student she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is the more likely alternative?a) Linda is a bank tellerb) Linda is a bank teller and active in the feminist

movement.

Answer:______


*Probability Rules: Multiplication Rule for Independent Events

• Multiplication Rule for independent events

When event A and Event B can occur together

Not A or B

A B

Joint probability = P (A and B)

P(A and B) = P (A) * P(B)


P(Bank Teller) > P(Bank Teller AND Feminist)

Discussion of Linda Question

• Nearly 90% of respondents choose the second alternative (bank teller and active in the feminist movement), even though this is logically incorrect

bank tellers feminists

feminist bank tellers

Junctions (“ands”) are always less likely than stand-alone statements.


Cognitive Illusions

• Kahneman & Tversky (1982)– “As the amount of detail in a scenario increases, its probability

can only decrease steadily, but its representativeness and hence its apparent likelihood may increase.”

– “The reliance on representativeness, we believe, is a primary reason for the unwarranted appeal of detailed scenarios and the illusory sense of insight that such constructions often provide.”

• Implications: consider a “rich” description of a reservoir depositional environment

• The description of Linda is more representative of a feminist bank teller so people, wrongly, conclude it is more likely that she is a feminist and a bank teller


High production days

• Daily production has been tracked for two oil fields for a year and the average production computed for each– The first oil field has 45 wells and the second 15.

• The number of days when production was above average for 60% or more of the wells in a field has been calculated.

• Which field do you think would have recorded more such days over the course of a year? a) The 45-well field.

b) The 15-well field.

c) Approximately the same (within 5%).


Insensitivity to Sample Size Results – Industry Respondees

0

10

20

30

40

50

60

70

Larger Smaller Equal

No o

f Par

ticip

ants

Answer


Insensitivity to Sample Size

• This question requires intuitive judgement as no data for calculation is available

• Central Limit Theorem:– The smaller the sample, the more likely it is to be a

deviant sample -> one with a mean and standard deviation significantly different from those of the population from which the sample was drawn

– Answer b, the 15-well field, is the most likely


Insensitivity to Sample Size

• People tend to ignore sample size they are dealing with when drawing conclusions from it.

• This is not consistent with the Central Limit Theorem which describes the changing characteristics of samples according to their size.

• Example – which is more likely:– Getting 6 heads in 10 flips of a coin, or

– Getting 6000 heads in 10000 flips of a coin.


Sequences - Representivity Heuristic:The “Law of Small Numbers”

• I have just tossed a fair coin 7 times. You have not seen the result

• You are invited to play a betting game to guess which of the three sequences below is the one I actually observed.

• Which sequence would you bet on

a) HHHHTTTb) THHTHTTc) TTTTTTT

• Using multiplicative rule for independent eventsP(A&B&C&D.) = P(A)*P(B)*P(C)*P(D) ….

P = (1/2)7 = 1/128P = (1/2)7 = 1/128P = (1/2)7 = 1/128

• ALL sequences have the SAME probability and are thus EQUALLY likely (or equally rare!)


The “Law of Small Numbers” (Tversky & Kahneman)

• We expect to see the same behaviour in small sequences that we would observe in large sequences – the mathematical “Law of Large Numbers” informs us of

behaviours that are approximately true for large sequences, and rigorously true for sequences near to infinity

• Some sequences are seen as more balanced or more “typical” and are thus thought to be more probable. Typicality is mistaken for probability.– with the result that we over-estimate probability


Sequences of Events, 2

• You are playing a game with a fair coin:– Heads you win $10, Tails you loose $10 (EV=$0)

• You have played 10 times so far and have had a luckystreak of 8 wins and 2 losses, so you are up $60– What is your expected value if you play a total of 1000 times

• You have played 10 times so far and have had an unlucky streak of, 3 wins and 7 losses, so your cumulative position is down $40– What is your expected outcome if you play a total of 1000 times


*Sequences of Events are not self-correcting :The “Gamblers Fallacy”

• You are playing a game with a fair coin:– Heads you win $10, Tails you loose $10 (EV=$0)

• You have played 10 times so far and have had a luckystreak of 8 wins and 2 losses, so you are up $60– What is your expected value if you play a total of 1000 times

– $60

• You have played 10 times so far and have had an unlucky streak of, 3 wins and 7 losses, so your cumulative position is down $40– What is your expected outcome if you play a total of 1000 times

– $-40


Sequences of Events are not self-correcting :The “Gamblers Fallacy”

• After holding bad cards on ten hands of poker, the poker player believes he’s “due” for a good hand.

• After winning $10,000 in the lottery, a woman changes her regular lottery number – after all, how likely is it that the same number will come up twice?

• The “hot hand”– If your favourite player has made his last four shots, is the

probability of making the next shot higher, lower, or the same as the probability of his making a shot without the preceding four hits?

• After 6 successive failures in an exploration play with a 30% chance of success “the next well is bound to be a discovery”

• Tversky and Kahneman:– “Chance is commonly viewed as a self-

correcting process in which a deviation in one direction induces a deviation in the opposite direction to restore the equilibrium. In fact, deviations are not corrected as a chance process unfolds, they are merely diluted.”


The Law of Small Numbers –We are not consistently inconsistent.

• The Law of Small Numbers is ubiquitous in the world of business decision-makers, where it lends unfounded credibility to the claims of those who have been successful for a few years in a row.

• An employee who does well several years in a row is surprised if performance is thought to be mean-reverting.


Probability Concepts: Multiplication Rule• The case of two dependent events

can be generalized for many events to

• If events are independent

which can be generalized for many events to

P(A and B) = P (A) * P(B|A)

P(A and B and C..) = P (A) * P(B|A) * P(C|BA)…

P(A and B and C…) = P (A) * P(B) * P(C) …

P(A | B) = P (A) and P(B | A) = P (B)

P(A and B) = P (A) * P(B)


Using P50s to assess completion times

• Suppose a project is made up of five, parallel, independent tasks, all of which must be completed for the project to be completed.

• You have estimated the P50 time for completion of each task to be 6 weeks.

• What is the chance that the project will be completed within 6 weeks?

• By definition, the probability of each task being completed on or before the P50 time (6 weeks) is 50%. Since the tasks are independent, then,

P(All 5 completed by 6 weeks) = 0.55 ~ 3%


Reliability of Predictors

• Historical estimates suggest one in every 1000 blow-out preventers (BOPs) has serious cracks.

• Suppose x-ray analysis is a very good, but not perfect, detector of these cracks.– If a BOP has cracks, x-rays will correctly say it has them

99% of the time– If a BOP does not have cracks, x-rays will wrongly say

that it has them 2% of the time

• A BOP has been x-rayed at random and the result was positive!– What is your intuitive estimate of the chance that is has

cracks?

Estimated Answer %


Use Probability Rules: in this case, Bayes’ Rule

• Given a collection of n mutually exclusive and collectively exhaustive events B1, B2, …. Bn and another event A

(likelihood) * (prior probability)posterior probability =

{(likelihood) * (prior probability)}

• The denominator (summation) is the total probability of A, that is, all the ways that A can occur and P(A|Bi) is called the likelihood function

P (A|Bi) P(Bi) P (A|Bi) P(Bi) P(Bi|A) = = nP (A) P (A|Bi) P(Bi)

i=1


What’s the relevance?

We make extensive use of conditional probabilities, and their formulation as Bayes’ Rule, to answer questions such as

How should prior probabilities be revised in the light of new information?

- e.g. how should we revise initial CoS estimate as a drillingprogram progresses

What is the value of taking 3D Seismic?

What is the value of adding extra slots to a platform in case OOIP is higher than thought?

What is the value of coring a well?


Insensitivity to Base Rate Results

0

10

20

30

40

50

60

70

80

90

0 20 40 60 80 100

Estimated Posterior Probability

No

of P

artic

ipan

ts


Uncertainty Propagation

Input 1

Model Decision Objective

AssessedUncertainties

Input n

Decision Criterion

PropagatedUncertainty Single Value

For non-linear models with uncertain inputs: the correct value of the decision criterion must be calculated by propagating the input uncertainty through to the decision objective uncertainty

eg NPV E[NPV] Equations(not brain!)


Averages don’t always work - misplaced

• For non-linear processes,– reservoir simulation– volumetrics with cut-offs– development alternatives

even if only a single, “best” estimate is required, we still need to use complete range of inputs - cannot use an average input

• Also, P10 (P90) results are NOT given by taking P10 (P90) inputs and running them through the model

ResultY = 4

ModelY = X2 Z2/

3 21 1 8

Mean = 10 Mean = 5

x z

Y

Simulation Result

0 30

True Mean~ 7.8


The Flaw of Averages:

• Unless model (f) is linear (“model” is a calculation)

where x,y,z,… are uncertain quantities, and

f is a calculation using x, y, z ....

average of f( x,y,z,…. )

f( average(x), average(y), average(z),….)


Flaw of Using Averages (after Savage)

NCF, Log, ...


Insensitivity to Base Rate: Frequency view

Applying the following reliabilities (CONDITIONAL probabilities)

P(T|R) Has Has NotHas 99% 2%

Has Not 1% 98%Test says

Real world

100% 100%

Has Has NotHas 99 1998

Has Not 1 97902Test says

Real worldto a sample of 100,000 people gives the following frequencies

100 99,900


Outline



Situations– Uncertainty Propagation– Updating Estimates for New Information– Complexity



*


Change Detection and Blindness

• People fail to notice large changes to visual arrays and scenes if they are briefly occluded

– e.g., during saccades, eye blinks, or presentations using simple masks

– the task is called ‘change detection’

– the inability to perceive change is called ‘change blindness’

• The previous simple example works for about a third to a half of people, and was created by Mondy and Coltheart (2002)


Counting Passes:Basketball movie

Basketball Movie


The Monty Hall Problem

• US TV Game show in early 1970s: “Let’s Make a Deal!”

– Show host Monty Hall

• Later discussed in newspaper column by Marilyn vos Savant

– Created an enormous response

– More than 10000 letters (many from professional mathematicians) denouncing her answer.

– Discussed in “The American Statistician.”



1/3 1/3 1/3There are three doors. One of these doors contains a prize. The other two do not. Therefore, the probability that any one of the doors contains the prize is 1/3.

You choose one door, say, door A.

A B C



1/3 1/3 1/3There are three doors. One of these doors contains a prize. The other two do not. Therefore, the probability that any one of the doors contains the prize is 1/3.

You choose one door, say, door A.

A B C

Given the option, should you stick with your original choice of door A, or switch to door B? Or does it matter?

I open one of the two remaining doors, say, door C, and reveal to you that it does not contain a prize.


*Solution: The Monty Hall Problem

1/3 2/3

A B C

Not behind A= 2/3

• If your original guess was correct and you switch, you lose.

• If your original guess was wrong and you switch, you win.

• Since your original guess would be wrong two out of three times, if you switch you’ll win two out of three times.


Use Probability Rules: in this case, Bayes’ Rule

• Conceptually, Bayes tells you how to update your prior probability of some event occurring (or model being true) after getting some information/data

based on the likelihood (probability) of observing the information/data, giventhat the event/model is true

(likelihood) * (prior probability)posterior probability =

{(likelihood) * (prior probability)}

• The denominator (summation) is the total probability of the data/information, that is, all the ways that it can occur


The Monty Hall Problem: Bayes’ Theorem

( | ) P(X and Y) P(X |Y) P(Y)P Y XP(X) P(X)

( and ) ( ) ( | )( | )P( ) P( )

P B open C P B P open C BP B open Copen C open C

( and ) ( ) ( | )( | )P( ) P( )

P A open C P A P open C AP A open Copen C open C

Bayes’ Theorem

We want to know the probability of the prize lying behind A or B, given the host opens C


Remember: Decomposing Total (or Marginal) Probability

BAB1

B2

B3

P(A) = P(A&B1) + P(A&B2) + P(A&B3)= P(A|B1)P(B1) + P(A|B2)P(B2) + P(A|B3)P(B3)

nP(A) = P (A|Bi) P(Bi)i=1

In “Monty Hall”:

A = open CB1 = prize behind AB2 = prize behind BB3 = prize behind C


The Monty Hall Problem: Probability that host will open door C

The a priori probability that the prize is behind door A, B, or CP(A) = P(B) = P(C) = 1/3

The probability that host opens door C if the prize is behind A:

The probability that host opens door C if the prize is behind B:

This image cannot currently be displayed.

P(open C|A) = 1/2

P(open C|B) = 1

The probability that host opens door C if the prize is behind C:P(open C|C) = 0

The total probability for host opening door C is then

P(open C) = P(A)*P(oC|A) + P(B)*P(oC|B) + P(C)*P(oC|C)= (1/3)*(1/2) + (1/3)*(1) + (1/3)*(0) = 1/2


21 movie

21 Movie


You Choose

Prize Behind

Host opens

Outcome (no Switch) Switch to Outcome

A A B or C Win B or C LoseA B C Lose B WinA C B Lose C WinB A C Lose A WinB B A or C Win A or C LoseB C A Lose C WinC A B Lose A WinC B A Lose B WinC C A or B Win A or B Lose

*Alternative Solution:List all possibilities

When you switch, you win 2/3 of the time- if you stick you win 1/3 of the time


Some of the things the letters said

• “I’m very concerned with the general public’s lack of mathematical skills. Please help by confessing your error.”

» Robert Sachs, PhD, George Mason University

• “There is enough mathematical illiteracy in this country, and we don’t need the world’s highest IQ propagating more. Shame!”

» Scott Smith, PhD, University of Florida

• “I’m in shock that after being corrected by at least three mathematicians, you still do not see your mistake.”

» Kent Ford, Dickinson State University


Some of the things the letters to vos Savant said:

• “…Your solution is the correct one and any REAL mathematician can produce a proof of its correctness. REAL mathematicians consider this a trivial problem… WHAT discipline do these respondents have their PhDs in? Is it adolescent behavior? If it is in mathematics, my second question is what institution granted it?”

» Professor Stephen J. Turner, Babson College



• “In a recent column, you called on math classes around the country to perform an experiment that would confirm your response to a game show problem. My eight-grade classes tried it, and I don’t really understand how to set up an equation for your theory, but it definitely does work!”

» Pat Gross, Ascension School, Missouri



• “After considerable discussion and vacillation here at the Los Alamos National Laboratory, two of my colleagues independently programmed the problem, and in 1 million trials, switching paid off 66.7 percent of the time. The total running time on the computer was less than one second.”

» G.P. DeVault, PhD, Los Alamos National Laboratory


Outline



Situations– Uncertainty Propagation– Updating Estimates with New Information– Complexity



Develop Strategyand SetFinancial Goals

Manage theBalance Sheet

Manage HumanResources

Use ProjectFinance for Middle East Build LNG

Capability inAlaska

Achieve 60/40Oil/Gas PortfolioSplit

Assume Operatorshipin CO2 Flood

Produce in theRockies

Develop VenezuelanHeavy Oil

Explore inColombia

Partner with XXin Deepwater

Trade GOM Shelffor Indonesia

Sell SouthLouisiana

Buy More Blocksin Viet Nam

A generic US Independent Executive Decision-Maker

• Complex problem-7 chunks of data

• “Optimum” is not intuitive-models can help


*Intuition: Steel Band Problem

• Assume the earth is perfectly smooth and a band of steel is places around the equator, which has a circumference of 50,000 km.

• Now we add in an extra 10m of steel, which slightly forces the band off the surface of the earth.

• What is your intuitive estimate of how much?

• Answer: 1.6 metres


Imagine two one-mile long pieces of railroad track, put end to end, and attached to the ground at the extremes. When it gets hot, each track length expands by one inch, forcing it to rise above the ground.

How high is the track off the ground at peak?

Give a high and low estimate such that you are 90% sure the correct answer lies between them.

Geometry Problem

x?

Ref: Richard Thaler

1 mile + 1 inch

1 mile


• Give a high and low estimate such that you are 90% sure the correct answer lies between them.

• Typical Answers:– Median Low Guess: ½ Inch

– Median High Guess: 2 Inches

– Ranges containing True Value: 15.9%

*Intuition: Geometry Problem

Ref: Richard Thaler


*Normative Rule

• Pythagorean Theorem2 2 2 2 2z x y x z y

2 2

2 2

x (1 m ile 1 in ch ) (1 m ile )

(5, 2 8 0 * 1 2 1) (5, 2 8 0 * 1 2 )

• Descriptive reality:– Most people underestimate x

– Why: We anchor on 1 inch and adjust insufficiently

1 2 6, 7 2 1 3 5 5 .9 8 in ch es 2 9 .6 fee t!

Ref: Richard Thaler


Intuition is Overrated

• Many decision-makers believe that intuition, repeated experience and their general intelligence will see them through

• Human beings are imperfect information processors

• We can’t always trust our intuition and perception. particularly in an uncertain environment!

• We need to use the appropriatetools and frameworks to address the uncertainties and decisions.


Intuition

• Intuition can be useful when– It is educated by making the same kind of decision

multiple times and observing the outcome – that is, when you are well calibrated

– The possible outcomes of the decision are not important

• Intuition can lead you astray when– Even slight degrees of complexity are involved– Uncertainty is involved, particularly if there are multiple

uncertainties which you have to combine to reach a conclusion

– Or when it is trained in one situation but then applied to another


When to trust intuition

• Daniel Kahneman (Heuristics & biases, prescriptive DA) and Gary Klein (naturalistic decision-making) debated when you can trust intuition or gut-instinct. They agreed that to protect decisions against bias, four tests should be passed:

– The familiarity test: Have we frequently experienced identical or similar situations?

– The feedback test: Did we get quick & reliable feedback on the outcomes of past decisions/judgments?

– The measured-emotions test: Is our thinking clouded by emotions we have experienced in similar or related situations? (“no”= pass!)

– The independence test: Are we likely to be influenced by any inappropriate personal motivations or biased thinking (“no”= pass!)

• If a situation fails even one of these four tests, we need to strengthen the decision process to reduce the risk of a bad outcome.

After a McKinsey Quarterly article (May 2010) based on “Conditions for intuitive expertise: A failure to disagree”, American Psychologist, Sept 2009


*Probability Training

• In terms of practical applicability, probability theory is comparable with geometry;

– both are branches of applied mathematics that are directly linked with the problems of daily life.

• While most people have a natural feel for geometry (at least to some extent), many people clearly have trouble developing a good intuition for probability.

• In no other branch of mathematics is it so easy to make mistakes as in probability theory.

– Conditional probabilities, and Bayes theorem in particular, are especially difficult


Probability Training

• In terms of practical applicability, probability theory is comparable with geometry;

– both are branches of applied mathematics that are directly linked with the problems of daily life.

• While most people have a natural feel for geometry (at least to some extent), many people clearly have trouble developing a good intuition for probability.

• In no other branch of mathematics is it so easy to make mistakes as in probability theory.

– Conditional probabilities, and Bayes theorem in particular, are especially difficult

“The theory of probabilities is at bottom nothing but common sense reduced to calculus; …

It teaches us to avoid the illusions which often mislead us;

… there is no science more worthy of our contemplations nor a more useful one for admission to our system of public education.”

Laplace – Theorie Analytique des Probabilites


Developing subjective Probabilities

Have you made repetitive forecasts of such an event in the past

Did you receive timely feedback on the accuracy of your forecasts

Is there a reference class of events that is similar and on which relative frequency information exists

Go ahead and assess subjective probability

Use the relative frequency information as subjective probability

Beware of impact of potential errors due to biases and inappropriate heuristics


Eliciting Subjective Probabilities

• The de Finetti game:– Bruno de Finetti

u An Italian statistician (1906 – 1985)

u Worked in the middle ground between mathematics and psychology

– A device to objectively measure subjective probability

• Most people ”lie” about probability without even being aware of it – they even lie to themselves.



• Suppose your friend just took an exam and feels good about how she did.

• She might tell you:– ”I aced it; I’m one hundred percent sure I’ll get a

perfect score.”

• The de Finetti game is a way to measure how sure she really is about having aced the exam.

• We need to ask your friend a series of questions to assess her true subjective probability of having aced the test.



• Tell you friend the following:– ”Let’s play a game. You have a choice.

1. you can either draw a ball from a bag that has 98 red balls and 2 black balls. If you happen to draw a red ball, I will give you one million dollars, or

2. you can decide to wait to see how you did on the exam, and if you receive a perfect score on that exam I will give you one million dollars.

– What’s you choice: draw or wait?”



• If your friend says:– ”Draw from the bag.”

– She then isn’t 100% sure that she has aced the exam – actually she must be less then 98% certain that she has aced the test if she chooses to draw from the bag.

• Now you ask the next question:– ”Now there are 80 red balls in the bag and

20 black ones.”

– ”Do you want to draw, and if you obtain a red ball get a million dollars, or wait to see how the exam went?”



• If your friend says:– ”Wait for the exam result.”

– Then you know that he is more than 80% sure that she really aced the test.

• Now choose a value in between, such as 90 red balls and ask:– ”Now there are 90 red balls in the bag and

10 black ones.”

– ”Do you want to draw, and if you obtain a red ball get a million dollars, or wait to see how the exam went?”



• At some point, say at 83 red balls (83%), your friend may say:– ”I’m indifferent between drawing a ball and waiting

for the exam result.”

• This (83%) is then her subjective probability of having aced the test.


Perceptual Limitations – visual illusions are a metaphor for cognitive illusions

• Previous examples were Cognitive/Probability Illusions

• Cognitive biases or illusions are similar to optical illusions in that the error can remain compellingeven when one is fully aware of its nature.– Awareness of the bias, by itself, does not

necessarily produce a more accurate perception.

• Cognitive biases, therefore, are also exceedingly difficult to overcome.


What Does This Have to Do With You and Me?

• These biases, which are well known to psychologists, have been shown to influence real decision-making behaviour.

• Evidence confirms oil & gas professionals are as prone to these biases and illusions as anyone else.

• In any decision-making situation it pays to pause and ask a few key questions– What are the non-intuitive factors might be present, particularly with

respect to uncertainty assessment and probability

– Am I motivated to see things a certain way?

– What expectations did I bring into the situation?

– Would I see things differently without these expectations and motives?

– Have I consulted others who don’t share my motives and expectations?


De-biasing

• Create awareness of the problems & recognition of situations when they may be present

– Anchoring– Overconfidence– Availability, Vividness & Recency

• Don’t rely on memory alone – write lists

• Actively challenge ourselves.– Stop to consider reasons why your judgement might be wrong.

• Abandon false comfort of single-point predictions.– Use ranges instead of single-point estimates.– Use multiple anchors.

• Calibration.– Feedback and accountability.


Understanding the Limits of Our Knowledge

• To know that we know what we know, and that we do not know what we do not know, that is true knowledge

Confucius

“It’s not what we don’t know that gets us into trouble, it’s what we know that

ain’t so”Will Rogers


The Pioneers

• Kahneman and Tversky– 2002 Nobel Price in Economics

• Russo and Schoemaker– Decision Traps– Winning Decisions

• Thaler– The Winner’s Curse

• Bazerman– Judgment in Managerial Decision Making

• Plous– The Psychology of Judgment and Decision Making

uncertainty, decisions, models & people - ausimm · industry performance: ... portfolio...

Documents