Department of Surgery and CancerImperial College London

20 May 2014

Norman Fenton

Queen Mary University of London and

Agena Ltd

Improved Medical Risk Assessment and Decision-making

with Bayesian Networks

Overview

• Why Bayes?• Why Bayesian networks?• Why NOT learn the models from data only?• Case study• Challenges and conclusions

1. WHY BAYES?

The Harvard ProblemOne in a thousand people has a prevalence for a

particular heart disease. A test to detect this disease has:• 100% sensitivity• 95% specificity If a randomly selected person tests positive what is the probability that the person actually has the disease?

Bayes Theorem

E(evidence)

We now get some evidence E.

H (hypothesis)

We have a hypothesis H with prior probability P(H)

We know P(E|H) but we want the posterior P(H|E)

P(H|E) = P(E|H)*P(H) P(E)

P(E|H)*P(H)P(E|H)*P(H) + P(E|not H)*P(not H)

=

1*1/1000

1*1/1000+ 5/100*999/1000P(H|E) = =

0.001

0.001 + 0.049950.0196

Waste of time showing this to most people!!!

Slide 6

Imagine 100,000people

Slide 7

Out of whom100 has thedisease

Slide 8

But about 5% of theremaining99900 peoplewithout thedisease testpositive.That is 4995 people

Slide 9

So 100 out of 5095 who testpositiveactually havethe disease

That’s justunder 2%

That’s very different fromthe 95% assumed by most medics

Total people100,000

1/1000

999/1000

Have the disease100

Don’t have the disease

99,900

So 100 out of 5,095who test positive actuallyhave the disease, i.e. under 2%

Test positive100

Test negative0

Test positive4,995

Test negative94,905

100%

0%

5%

95%

2. WHY BAYESIAN NETWORKS?

A Simple Bayesian Network

..but here is a typical

causal model

Calculations from first principles are

infeasible and incomprehensible

Actual model in medical negligence case

This model already reaches limit of comprehensibility for

manual calculations and event trees

• MRA• CA

• Ischaemic• Small aneurysm• Large aneurysm• CSP

Detected by Test9,900

Undetected by Test100

Detected by Test90

Undetected by Test10

Detected by Test0

Undetected by Test10,000

Die from burst/bleeding

Die from CSP

99%

1%

90%

10%

50%

0%

100%

2%

2%

CA Test PathwayCause of Palsy Test Result Outcome Deaths

2

0

5000

5002TOTAL

= 1.495%

1

14,952 out of 1,000,000 give risk

Stroke

Strokes

Don’t die

99

Stroke

Stroke

Die from burst/bleeding

Don’t die Stroke

Don’t die Stroke

1%

1%

1%

1%

1%

50%

98%

98%

99

2

981

1

1

0

10 0 0

5000

500050

1%Stroke

50

97999799

9950

Total people1,000,000

Large9,900

Small100

CSP10,000

Others (ischaemic)980,000

1%

1%

1%

98%

Aneurysm10,000

99%

Total people1,000,000

Large9,900

Small100

CSP10,000

Others (ischaemic)980,000

Detected by Test9,405

Undetected by Test495

Detected by Test50

Undetected by Test50

Detected by Test9,000

Undetected by Test1000

Die from burst/bleeding

Die from burst/bleeding

Die from CSP

Die from CSP

1%

1%

1%

98%

95%

5%

50%

50%

50%

90%

10%

2%

2%

20%

MRA Test PathwayCause of Palsy Test Result Outcome Deaths

10

1

1800

500

2311TOTAL

= 0.2311%

0

0

0

2311 out of 1,000,000 give risk

Aneurysm10,000

99%

Much better solution…use a Bayesian Network tool

Computation for Catheter Angiogram

Mean:9950

Mean:5002

Computation for MRA Scan

Mean:0

Mean:2311

The Calculator Analogy

No need for p-tests or classical confidence intervals

• Drug “Precision” weight loss: Everyone in trial lost between 4.5 and 5.5 pounds

• Drug “Oomph” weight loss: Everyone in trial lost between 10 and 30 pounds

• Which drug can we ‘accept’, i.e. reject null hypothesis of ‘no weight loss’?

• Classical stats provides nonsensical answers

No need for p-tests or classical confidence intervals

3. WHY NOT LEARN THE MODELS FROM DATA ONLY?

A typical data-driven study

Age Delay in arrival

Injurytype

Brain scanresult

Arterialpressure

Pupildilation

Outcome (death y/n)

17 25 A N L Y N

39 20 B N M Y N

23 65 A N L N Y

21 80 C Y H Y N

68 20 B Y M Y N

22 30 A N M N Y

… … … .. … …

Delay in arrival

Injurytype

Brain scanresult Arterial

pressure

Pupildilation

Age

Outcome

A typical data-driven study

Purely data driven machine learning algorithms will be inaccurate and produce counterintuitive results e.g. outcome more likely to be OK in the worst scenarios

Delay in arrival

Injurytype

Brain scanresult Arterial

pressure

Pupildilation

Age

Causal model with intervention

Dangerlevel

Outcome

TREATMENT

..crucial variables missing from the data

Determining drug effectiveness

Basic results for drug effectiveness

Drug AThe mean financial benefit is $4156

Drug BThe mean financial benefit is $2777

Ban drug B?

Model with latent variable (same data)

Note that most patients in the sample had minor

case of the condition

…and most patients were given drug A

Results with 'Patient condition' major

Drug B30% positive outcome.The mean financial benefit is $1000

Drug AOnly 10% positive outcome.The mean financial benefit is $400

OK, so we might need expert judgment when we have missing data, but with good experimental design and lots of good quality data we can surely remove dependency on experts ……

A machine learning fableA and B are two medical conditions very well known to doctors Bill and Ludmila. These conditions are pretty rare (both have an incidence of about one in 1,000 people). There is a third medical condition C (whose name is “FiroziliRalitNoNeOba”) that Bill has heard the name of, but knows nothing about. But Bill has heard that patients with either A or B usually also have C. Bill has a massive database of 600,000 people with the details of which conditions they have.

Bill’s dataPatient number A B C 1 No No No 2 No No No 3 Yes No Yes 4 No No No 5 No No No 6 No No No 7 Yes No Yes 8 No Yes Yes 9 No No No 10 No No No 11 No No No 12 No Yes Yes 13 No No No 14 No No No …. … … …. … … 600,000 No No No

Bill’s machine learning mate FredCan use this database to ‘discover’ the underlying causal model (Bayesian Network) relating A, B, and C. But Ludmila says she knows the correct model without data:

Fred warns against this

She also “knows” the probability tables

Fred’s learnt model

• Ludmilla disagrees with the last column of table C • Fred: “Not enough data for that”• Bill: “…why can’t we simply conclude that C must be true when

both A and B are?”

600 out of 600,000 have condition A

600 out of 600,000 have condition B

Every single person with condition A also has C and every single person with B also has C.

Ludmilla’s knowledge

• The name of Condition C - FiroziliRalitNoNeOba - is actually a Russian word.

• Its literal translation is:– ‘A person suffering from either Firoz or Ralit but

not both’. – ‘Firoz’ is the Russian word for condition A and

‘Ralit’ is the Russian word for condition B.”

Moral of the story

• Sometimes you have to trust experts to provide more informed quantitative judgement than you will get from data alone.

• Even really big datasets will be insufficient for some very small problems.

• Trusting the expert can save you a whole load of unnecessary data-collection and machine learning effort.

4. CASE STUDY

Trauma Care Case Study• QM RIM Group

– William Marsh– Barbaros Yet

• The Royal London Hospital– Mr Zane Perkins– Mr Nigel Tai– ACIT Data

• US Army Institute of Surgical Research– Lower Extremity Injury

DataYet, B., Perkins Z., Fenton, N.E., Tai, N., Marsh, W., "Not Just Data: A Method for Improving Prediction with Knowledge", Journal of Biomedical Informatics, 2014 Apr;48:28-37

BN v MESS Score

• Prediction: coagulopathy, death (c.f. GCS, TRISS)• Flexible inputs• Patient’s physiological state

– Causal modelling: informed by knowledge

How the BN Model Differs

Life Saving: Prediction of Physiological Disorders

Limb Saving: Prediction of Limb Viability

www.traumamodels.com

5. CHALLENGES AND CONCLUSIONS

Challenges• Apparent paradox on using experts• Expert systems have a bad reputation• Resistance to subjective priors• Building new large-scale BN models, especially

with minimal data• Interacting with large-scale BN models• Explaining the results of BN models

BAYES-KNOWLEDGE (Effective Bayesian Modelling with Knowledge Before Data)www.eecs.qmul.ac.uk/~norman/projects/B_Knowledge.html

Conclusions (1)

• Purely data driven approaches using Machine learning and statistics DO NOT WORK

• At best captures what did happen Vs what would have happened

• Need to move to data + knowledge approach• BNs provide the key

Conclusions (2): BN Benefits

• Data + knowledge• Models uncertainty and causality• Predictions and diagnosis• Avoid medical statistics fixation on p-values

and confidence intervals• Incorporate qualitative and quantitative

variables• Identify causal effects without RCTs• New generation expert systems

Blatant Plug for Book

CRC Press, ISBN: 9781439809105 , ISBN 10: 1439809100