Command Post of the FutureLimited Objective Experiment-1

Presented to:Information Superiority Workshop II:

Focus on Metrics28 - 29 March 2000

Presented by:Dr. Richard E. Hayes

Evidence Based Research, Inc.1595 Spring Hill Road, Suite 250

Vienna, VA 22182703-893-6800

www.ebrinc.com

2

Agenda

• HEAT and Quality of Information

• CPOF and Quality of Awareness

• Summary

3

Headquarters Effectiveness Assessment Tool (HEAT)

HEAT is a set of consistent procedures that measure the effectiveness of a military headquarters or command center

• Quantitative, objective, and reproducible assessment of quality of C2 processes and overall effectiveness of directives

• Applied more than 50 times to exercises and real world applications in more than 200 command centers

HEAT views headquarters as analogous to an adaptive control system that influences environment through plans (directives)

• Headquarters can be judged by viability of plans– Remain in effect for intended period– Include contingencies

• Diagnostic scores for quality of C2 processes

4

HEADQUARTERS

DEVELOP ALTERNATIVE ACTIONS

UNDERSTAND

MONITOR

PREDICT CONSEQUENCES

DECIDE

DIRECT

QUERYINFORM

ENVIRONMENT: - OWN & ENEMY FORCES - PHYSICAL - POLITICAL & ECONOMIC Awareness

HEAT Analytic Structure

5

HEAT Information Measures

Completeness• Are all the required elements present?

Correctness• Does the information match the truth?

Consistency• Is the information the same across all nodes?

Precision• Was the information presented at the proper level of precision?

Currency (Latency)• How old is the information?

6

The goal of CPOF is to shorten the commander’s decision cycle to stay ahead of the adversary’s ability to react.

Command Post of the Future

Command Post of the Future (CPOF) is a DARPA program that aims to:

• Increase Speed and Quality of Command Decisions

– Faster recognition and better understanding of changing battlefield situation

– Faster and more complete exploration of available courses of action

• Provide More Effective Dissemination of Commands

– COA capture for dissemination of commander’s intent

– Status and capability feedback from deployed operators

• Enable Smaller, More Mobile and Agile Command Structures

– More mobile, distributed command element

– Smaller support tail & reduced deployment requirements

7

Limited Objective Experiment - 1(LOE -1)

Hypothesis:

• Tailored visualizations will improve Situation Awareness

MOE:

• Correctness of Situation Awareness comprehension

• Quality of Pattern Recognition

• Level of Entity Retention

8

Force-on-Force

Insurgency

Situation 13Situation 13Situation 10Situation 10

Situation 5Situation 5Situation 4Situation 4

Less Complex

More Complex

LOE-1 Scenario Space

9

Data Collection

• 157 valid runs over 5 days (98% of design goal of 160)

• Data Collection Vehicle - Debrief Questionnaire

• Questions divided into four major elements: overall Situation Assessment, Patterns, Elements, and a Sketch of the situation

• Military experience data collected for each subject

• Run randomization compensated for variations between coders and subjects

10

Top Level Measures

• Overall Situation Awareness

– Picture of the battlespace - enemy posture and intentions, friendly possibilities for favorable action

• Pattern Recognition

– Dynamics of the battlespace - relative positions, strength, and capabilities

• Element Retention

– Locations, status, events over time

• Ability to draw a general Sketch with relative positioning of important entities in space and time

11

1.You have three minutes to brief your commander on your assessment of the situation, what do you tell him?

1a. In this situation what are the friendly:

Opportunities?Risks?

1b What are the adversary’s:

Offensive capabilities?Defensive capabilities?Vulnerabilities?Intentions?

1c. What are the vulnerabilities for own forces?

2a. If you are not certain about adversary intentions and capabilities, what are the different possibilities?

2b. If you are not certain, what information would you require to resolve this uncertainty?

3a. What specific elements or element combinations lead you to your situation assessment?

3b. Please make a sketch of the situation, including the specific elements or element combinations cited in your previous answer.

3c. Was any information you would find valuable missing from the presentation of the situation?

4. What was good and or bad about this display? What would you like to see added, removed or changed?

Debrief Questionnaire

12

Metrics

Overall• Prompted % of key components wrong inferences added inferences

• Unprompted % of key components wrong inferences added inferences

Patterns % of pre-identified wrong added

Elements % of important elements wrong added

Sketch % of pre-identified elements wrong added

temporal pattern

geographic pattern

13

Interpretation• CPOF Technologies Generated:

– Better Situation Awareness (Higher Mean or X)– CPOF Technologies Performance Improves for Prompted

CPOF Technologies SignificantlyOutperform Control in Overall Scores

Prompted

23.8923.89 25.6225.62

21.4121.41

33.8633.86

N=157 p=.007

Unprompted

22.3022.30 23.4023.40

17.77 17.77

25.8025.80

N=157 p=.058

ss

xx

ControlCPOF Technologies

14

CPOF Technologies SignificantlyOutperform Control in Complex Situations

Interpretation:

CPOF Technologies generated better Situation Awareness (higher mean or x) in complex situations

Unprompted Prompted

6.286.28 18.5318.53

3.953.95

17.2417.24

N=78 p=.000

8.858.85 18.2818.28

5.105.10

19.9619.96

N=78 p=.000

sControlCPOF Technologies

x

15

12.0612.0615.0015.00

Treatment B Outperforms Treatment A in Overall Situation Awareness in Situation 13

Unprompted

15.9315.93

22.93

12.2112.21

N=29 p=.073

Prompted

14.8514.85

29.6629.66

12.71

N=29 p=.002

s

x

Treatment BTreatment A

Interpretation:

Treatment A used icon visualization scheme that subjects stated was confusing

16

• CPOF: “Present timeis reached.”

Bn - enemy

Co - enemy

Pt - enemy

Bn - friend

Co - friend

Pt - friend

Legend

Today’s Technology

Allegiance Color Coding

Treatment B

Blobs and Allegiance Color Coding

Treatment A

Function Color Coding

The Force-on-Force Visualization Treatments

17

Summary

• Information does not equal awareness

• Natural layers of overall patterns and elements

• Measures must always be developed in a military context

• Overall

• Patterns

• Elements

• Incorrect

• Richness