The Value of Information

John Medhurst, Larrainzar Consulting Solutions Ltd

Maj Ian Stanton, Dstl

Ian Mitchell, Dstl

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Value of Information

• Seeks to answer the question – “What is the value of information in NBC defence”

• Work for this year

– Seek to understand one element of NBC decision making – the response to biological detector alarms

– Use of an experiment to find out how military personnel use information to decide whether a BW attack has occurred

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Experimental Method• Simple scenario and abstract, high-level map provided, with sensor

locations and rough location of own units

• Other information presented on a series of cards

• Key advantages of cards

– Allows us to present info in ‘bite sized’ chunks

– Allows us to ensure that information is received and understood

• Players required to decide when to declare ‘Precautionary Alert’ and when to declare ‘Probable Attack’. Corresponds to issuing of NBC 3 and decision to recommend initiation of medical countermeasures.

• Eight Serials with Latin Square design (8 x 8), order of serials assigned randomly from Latin Square

• Total of approx 550 data points if all sequences of cards accounted for – all serials carried through to end, even if decision made

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Context

Alarms

Indicators

Corroboration

Negative Info

Information Cards15/09/2003 - v10

Environment

Threat Level

Conflict Level

Consequences

TypeGeneric

Specific

Collection Method

Information

Missile Explosion

Artillery Attack

Locals Seen in Vicinity

Mortar Attack

Suitable Met Conditions

Low Flying Aircraft

Low Flying UAVs

Boats

Explosions

Clouds

Confirmatory Sensors

Detections by other forces

Illness Reported

Dead Animals

Illness among local population

Illness among civilian hangers-on

Claims by enemy

Action taken by other forces

CNN reports

Sandstorm

Mist

Lack of Assessed Capability

Unanticipated event

Low Tension

Poor Operator Training

High False Alarm Rates

Daytime

Wind blowing wrong way

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

G1

The Detector at Location A reports an alarm on the

ASAS/CFL

CO1

CNN reports that the King Mohammed Hospital is taking in many patients with a mysterious

flu-like disease

I1

One of the Forward Units reports enemy carrying out

ranging fire across the border

N1

Weather Reports indicates strong winds blowing from the

SE for the next 48 hours

What the

Cards Look Like

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Quantitative Results

• Analysed using multivariate Probit analysis

• Simple MV Probit Model

– Consists of

• An intercept for precautionary alert• An intercept for probable attack• A factor for each of the six card types

– Gives a value which can be looked up on the normal distribution to give a probability

– Intercepts can be interpreted in terms of Bayesian ‘prior belief’

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Predicted P vs Actual P - Precautionary Alert - Data and Coefficients from Probit Analysis

y = 0.8171x + 0.084R2 = 0.7839

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual P

Pre

dic

ted

P

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Predicted P vs Actual P - Probable Attack- Data and Coefficients from Probit Analysis

y = 0.6667x + 0.0706R2 = 0.6307

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual P

Pre

dic

ted

P

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Discussion

• Although the simple model ignores elements such as pair-pair variability, these elements are what the probabilistic element of a probit model is intended to represent

• The simple model accounts for 78% of the variation in P(precautionary alert) and 63% of that in P(probable attack)

• Key question is how good the model is at predicting behaviour

• To look at the accuracy of the simple model, we have applied the simple formula on a serial by serial basis

• A sample of the results (Serials 1-4 for Precautionary Alert) are shown on the next slide

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Probability of Prec Alert - Serial 1

0.00

0.20

0.40

0.60

0.80

1.00

G2 C2 N1 N7 G4 S14 I6

Card

P

Estimated

Actual

Probability of Prec Alert - Serial 3

0.00

0.20

0.40

0.60

0.80

1.00

C10 S12 I3 I9 I12 S9 N9 Y7

Card

P

Estimated

Actual

Probability of Prec Alert - Serial 4

0.00

0.20

0.40

0.60

0.80

1.00

I10 N8 G3 C11 I8 N11 G2 S6 S5 Y8

Card

P

Estimated

Actual

Probability of Prec Alert - Serial 2

0.00

0.20

0.40

0.60

0.80

1.00

Card

P

Estimated

Actual

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Effectiveness of the Simple Model

• To assist in using the game as a training aid a ‘DS Solution’ has been developed for the serials

• Accuracy of the simple model in predicting the ‘DS Solution’ is 75% if p>0.5 is used as the trigger

• Accuracy of median human pair is slightly less

• There is potential for a similar algorithm to be incorporated in the NBC BISA as a ‘wake-up’ system

• Could be calibrated by use of the game as a training aid – would allow a continually updated knowledge base to support decision-making in the field

© Dstl 2003

20 April 2023 Dstl is part of the Ministry of Defence

Conclusions

• The initial aim of this phase of the study has been achieved – we have established the relative value of information in responding to a possible BW attack

• There is potential for use of the experimental method as a training game, for which there is a demonstrable need

• We have a simple model of human response to information, and an experimental method for estimating the effect of individual pieces of information on a decision

• We also have a simple model of the response to biological alarms that has potential to be used as a decision support tool in the NBC BISA