ICFIS, Leiden

21 August 2014

Norman Fenton Queen Mary University of London and Agena Ltd

norman@agena.co.uk

Limitations and opportunitiesof the likelihood ratio approach

for evidence evaluation

Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, Volume 54, Issue 4, Pages 274–287, July 2014

Overview

1. Revisiting the Likelihood Ratio – first principles, theoretical benefits and limitations

2. Revisiting the LR in two well-known cases3. Why Bayesian networks are needed 4. Conclusions and way forward

1. REVISITING THE LIKELIHOOD RATIO – FIRST PRINCIPLES

Was Mrs Peacock the murderer?H: “Mrs Peacock guilty”

P(H) = 1/6

E: “The murderer was female

P(E | H) = 1

P(E) = 1/2

By Bayes P(H | E) = 1/3

When does evidence E support a hypothesis H?

When our belief in H increases as a result of observing E, i.e. when P(H | E) > P(H)So, suppose E supports H and that H’ is an alternative mutually exclusive hypothesis. Can we conclude that our belief in H’ must have decreased? NO!! (suppose H’ is “Miss Scarlett is murderer”)Except when H’ = not H

P(H’ | E) = 1 – P(H | E) < 1 – P(H) = P(H’)

A simple formal definition of probative value of evidence

The ratio R: • R > 1 means E supports H• R < 1 means E supports not H• R = 1 means E is neutral for H

Why do we never see this definition used?

Because of obsessive and irrational fear of the explicit

PRIOR P(H)

Instead the Likelihood Ratio is usedIn addition to H we have to consider an alternative

mutually exclusive hypothesis H’

L= By Bayes Theorem providing that H’=not H we can

conclude• L>1 if and only if R>1• L<1 if and only if R<1• L=1 if and only if R=1So L is a valid measure of the probative value of

evidence E for H (when H’ = not H)

Benefits of the LRSimple formula for probative value of evidenceNo need to explicitly consider prior for HForces forensic experts to consider the likelihood of both the prosecution hypothesis and the defence hypothesis

But note:It is meaningless to talk

about the Likelihood Ratio being a measure of the

probative value of evidence without explicit reference to

Bayes Theorem

And especially note…..When

H’ ≠ not H

the notion that the LR is a measure of probative value of evidence is tenuous and potentially misleading

When H’ ≠ not H…• Knowing that LR>1 just tells us that the ratio

of posterior probabilities (of H and H’) is greater than the ratio of prior probabilities

• So all we can conclude is E supports H more than it supports H’. But E may not support H at all because we can still have P(H|E) < P(H)

• Similarly LR=1 only tells us E supports both H and H’ equally. That does NOT mean E is neutral; P(H|E) might be very different to P(H)

Was Mrs Peacock the murderer?H: “Mrs Peacock guilty”

E: “The murderer was female

P(E | H) = 1P(E | not H) = 2/5

LR= 2.5

But if H’: “Miss Scarlet guilty”P(E | H’) = 1

LR=1

Issues and limitations of the LR• ‘probative value’ is not what people think it means

when H’ is different from not H• But it is difficult to work with exhaustive pairs of

hypotheses• Priors can never be truly ignored• Evidence E is rarely ‘simple’ – normally involves E1

and E2 that require separate likelihoods• Can be difficult even to avoid non-mutually exclusive

hypotheses in practice• Even if we get it all right LR of source level hypotheses

tells us NOTHING about LR of offense level hypotheses

ExampleFred and Joe live at the same address. Gun X is registered to that address. Bob is found murdered from a gun shot. Evidence E: “there is a gun in Fred’s house with FDR that matched that from the crime scene.” Fred is charged with the murder of Bob. The offence level hypotheses are: Hp: Fred fired the shot that killed Bob not Hp: Fred did not fire the shot that killed Bob

The source level hypotheses are:Hp1: Fred owned gun that killed Bob not Hp1: Fred did not own gun that killed Bob

Some reasonable assumptions

LR=1 for source level hypotheses…..but E has real probative value on Hp

Essentially irrelevant

Prior state of the BN

Calculating the probability of evidence E under the two values for H1p

P(E | not H1p) = 0.9891 (unchanged from prior)

P(E | H1p) = 0.9891 (unchanged from prior)

Evidence is observed

Probability of Hp jumps from 1% to over 9%

..the evidence is certainly not neutral

2. REVISITING THE LR IN TWO WELL-KNOWN CASES

R v Sally Clark 1999-2003

Convicted and ultimately cleared of murdering her 2 children

Sally Clark Revisited: A new issue in the probability experts’ reasoning

Hd : Sally Clark’s two babies died of SIDSHp : Sally Clark murdered her two babies

“(Prior) probability of Hd over 100 times greater than (prior) probability of Hp”“So assuming LR of 5 posterior of Hd still 20 greater

Hd : Sally Clark’s two babies died of SIDSHp : Sally Clark murdered at least one of her two babies.

(Prior) probability of Hd only 2.5 times greater than the (prior) probability of Hp

R v Barry George, 2001-2007

Jill Dando

R v Barry George (revisiting the Appeal Court judgment)

Hp: Hypothesis “BG was man who shot JD”E: “Single particle of FDR matching that from the gun that killed JD found in BG coat pocketDefence likelihood P(E|not Hp) = 1/100…But Prosecution likelihood P(E| Hp) = 1/100So LR = 1 and evidence ‘has no probative value’But the appeal transcript suggests a problem…

Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, Volume 54, Issue 4, Pages 274–287, July 2014

Confusion from experts about the hypotheses

• Not clear that Hp stated was really the same prosecution hypothesis considered by the experts – H1p: “The particle found in BG’s pocket came from a gun

fired by BG”.– H2p: “The particle found in BG’s pocket came from the gun

that killed JD”.

• Transcript suggests the experts did not adhere to the assumption that defence hypothesis Hd was simply “not Hp”, i.e. “BG was not the man who shot JD”.– H1d: “Integrity of BG coat was corrupted”

LR=1?

P(E | Hp) = P(E | H1d) but the evidence E is not neutral as concluded by expert and accepted by the court. It favours Hp.

3. WHY BAYESIAN NETWORKS ARE NEEDED

More comprehensive BN model needed in BG case

Target is type X

Target is source

Source is type X

Target tested X

Source tested X

Even single piece of forensic match evidence is NOT a 2-node BN

Source is type X

Bayesian nets: what we need to stress

Separate out assumptions from calculationsCan incorporate subjective, expert judgementCan address the standard resistance to using subjective probabilities by using ranges.Easily show results from different assumptions

…but must be seen as the ‘calculator’

The potential of Bayesian Networks

“I assert that we now have a technology that is ready for use, not just by the scholars of evidence, but by trial lawyers.”

Edwards, W. (1991). "Influence Diagrams, Bayesian Imperialism, and the Collins case: an appeal to reason." Cardozo Law Review 13: 1025-107

4. CONCLUSIONS AND WAY FORWARD

Summary• LR and probative value of evidence may not

be what people think it is• In isolation the LR may be highly misleading• Doing things correctly requires fuller models

- BNs• But Bayesian arguments cannot be

presented from first principles

Blatant Plug for Book

CRC Press, ISBN: 9781439809105 , ISBN 10: 1439809100