5/12/2012

1

Hypothesis Mapping as an Alternative to ACH

Dr. Tim van Gelder University of Melbourne

Principal, Austhink Consulting tvg@austhinkconsulting.com

5/12/2012

2

Analysis of Competing Hypotheses

“Classic” ACH Method

1. Start with a table or matrix. 2. Enter all relevant hypotheses

across the top of the table. 3. Enter all items of evidence down

the left side of the table. 4. For each item of evidence,

indicate its bearing (+,-, ?) on every hypothesis.

5. Refine the table, revising the hypotheses and setting aside “useless” (non-diagnostic) evidence.

6. Seek additional evidence, looking to disprove hypotheses.

7. Make assessment, focusing on negative evidence.

5/12/2012

3

Some strengths of ACH

• It is a structured analytic method (?)

• It is designed to counteract cognitive limitations - particularly confirmation bias (but does it really?)

• It uses a simple, familiar external representation (matrix)

• The matrix structure accommodates the many-many relationship between evidence and hypotheses

• It helps create a common/shared understanding of a problem

• It creates an “audit trail” of thinking

Some things I don’t know…

• To what extent is ACH actually used on the job in intelligence analysis?

• To what extent does ACH in fact improve/hamper on-the-job performance?

• To what extent has classic ACH been superseded by any better methodology?

5/12/2012

4

So I will:

• Focus on “classic” ACH

• Focus particularly on the use of a matrix

• Focus on problems inherent to use of a matrix structure

Whatever happened to Loretta Cohen? You’re a police investigator. You have inherited a Missing Persons case from someone else. This is the information gathered so far. Loretta Cohen works (or worked) as the PA to the Finance Manager at Patrick Stevedores. She was reported missing on Wednesday 3 May, after failing to turn up to work on Monday 1 May. Her colleagues pointed out that three lots of $50,000 have gone missing from the Finance Department over the last six months. Loretta’s Hyundai Excel was discovered parked near the cliffs at the surf beach. Loretta is an orphan and an only child. Friends at work say Loretta broke up with her boyfriend 6 weeks ago. They also reported that Loretta seemed to be upset by almost daily phone calls from him until about a week before her disappearance. Subsequent investigation has revealed that Loretta’s former boyfriend, 34 year old Mark Tyson, has a conviction for aggravated assault from ten years ago. When the previous investigator had a look at Loretta’s flat, nothing was obviously missing; but it appears Loretta was in letter correspondence with an Islamic activist claiming to be planning a suicide attack in Tel Aviv, and her personal diary described a recent sense of hopelessness. According to neighbors, a mysterious white van has been seen parked outside Loretta’s apartment block for long periods of time, with a man sitting inside it.

5/12/2012

5

H1 H2 H3 H4 H5 H6 H7

Killed by

work

colleague Suicide

Killed by

boyfriend

Ran away

to join

terrorists

Killed by

burglar

E1 Worked as PA to Finance

Manager + + ? ? ? E2 3 x 50K gone missing from

Finance Dept + + ? + ? E3 Loretta’s car parked near

cliffs at surf beach ? + ? ? - E4 An orphan and only child

? + ? + ? E5 Broke up with boyfriend 6

weeks ago ? + + + ? E6 Upset by daily phone calls

from boyfriend ? + + + ? E7 Boyfriend conviction for

assault ? ? + ? ? E8 Nothing missing from flat

? + + ? - E9 Correspondence with Islamic

activist planning suicide

attack ? + ? + ?

E10 Mysterious white van outside

apartment + ? + ? +

Problems with ACH

• Requires far too many discrete judgments

– Where many of those are pointless

• Oversimplifies evidence-hypothesis relationship

• Treats hypothesis sets as flat

• Treats evidence set as flat

• Discombobulating

5/12/2012

6

Requires too many discrete judgments

“The exercise we do is a two-day counter terror problem with 130 pieces of evidence. If students assess all evidence in order to attenuate selection bias, and they test 4 hypotheses, they are making 520 individual judgments. Ouch.”

H1 H2 H3 H4 H5 H6 H7

Killed by

work

colleague Suicide

Killed by

boyfriend

Ran away

to join

terrorists

Killed by

burglar

E1 Worked as PA to Finance

Manager + + ? ? ? E2 3 x 50K gone missing from

Finance Dept + + ? + ? E3 Loretta’s car parked near

cliffs at surf beach ? + ? ? - E4 An orphan and only child

? + ? + ? E5 Broke up with boyfriend 6

weeks ago ? + + + ? E6 Upset by daily phone calls

from boyfriend ? + + + ? E7 Boyfriend conviction for

assault ? ? + ? ? E8 Nothing missing from flat

? + + ? - E9 Correspondence with Islamic

activist planning suicide

attack ? + ? + ?

E10 Mysterious white van outside

apartment + ? + ? +

5/12/2012

7

Oversimplifies Evidence-Hypothesis Relationship

• Matrix structure can’t represent co-premise structures (assumptions, auxiliary hypotheses)

H1 H2 H3 H4 H5 H6 H7

Kill

ed b

y w

ork

colle

agu

e

Suic

ide

Kill

ed b

y b

oyf

rien

d

Ra

n a

way

to

join

terr

ori

sts

Kill

ed b

y b

urg

lar

E8 Nothing missing from

flat

? + + ? -

Treats hypothesis sets as flat

• Hypotheses are just listed across the top

• This neglects meaningful structure in the hypothesis set and creates unnecessary work

5/12/2012

8

Treats evidence set as flat

• Evidence items listed down the side

• No way to display sub-ordinate arguments & evidence

It’s discombobulating…

5/12/2012

9

Or more generally, ACH…

• Fundamentally Flawed Logically

– misconceives hypothesis sets, evidence sets, and the relationships between them

• Fundamentally Flawed Psychologically

– Decontextualising and Discombobulating

– Misconceives the nature of analytical expertise

Hypothesis Mapping

5/12/2012

10

Simple Hypothesis Mapping Method

1. Identify Issue(s). Write in a question box.

2. Develop Hypotheses. Add Hypothesis boxes to map.

3. Marshal Evidence. Add Evidence and Counter-Evidence boxes to map.

4. Evaluate Hypotheses. Identify favoured hypotheses.

5. Test Hypotheses. Seek new evidence challenging favoured hypotheses. Add to map.

6. Draw Conclusions. Rate plausibility of hypotheses.

Loretta Hypothesis Map (Simple)

5/12/2012

11

Loretta Hypothesis Map

Structured Hypothesis Set

5/12/2012

12

…allows evidence to be correctly located

And makes it visually transparent how much you’ve got and where gaps are

Structured Evidence Sets

5/12/2012

13

Co-premise/auxiliary hypotheses

Co-premise/auxiliary hypotheses

5/12/2012

14

HM Strengths

• Remedies many major problems of ACH

• Intuitive visual format

• Compatible with other methodologies

– Argument Mapping

– Decision Mapping

– Pyramid Principle

• Logic trail

Hypothesis Mapping Weaknesses

• Uses lots of space

• Evidence may need to be duplicated

• More dependent on custom software tools