Improving Collaboration in Unmanned Aerial Vehicle

Operations

March 27, 2007

Stacey D. ScottHumans & Automation Lab

MIT Aeronautics and Astronauticshttp://halab.mit.edu

Teamwork in Unmanned Aerial Vehicle (UAV) Teamwork in Unmanned Aerial Vehicle (UAV) OperationsOperations

Intelligence Consumers (e.g., Ground Troops)

UAV Operators

Lawyers & PoliticiansGround Crew

Shadow UAV

Supporting Futuristic UAV Teams Engaged in Supporting Futuristic UAV Teams Engaged in Time-Critical OperationsTime-Critical Operations

Future UAV Teams

Current Collaboration Tools Insufficient for Current Collaboration Tools Insufficient for Time-Critical OperationsTime-Critical Operations

• Increasing reliance on collaboration tools- e.g., email, instant messaging (chat)

• These tools introduce communication overhead, hindering management of tasking & team coordination (Scott et al., 2006)

• Trend in corporate and educational collaboration systems towards intelligently sharing activity information to improve planning & coordination in networked teams- Activity awareness design approach

(Carroll et al., 2006)

• Promoting activity awareness in UAV operator interfaces may:- help UAV teams coordinate their actions- reduce the costs of obtaining activity-

related information

Research Goals: Supporting Activity Research Goals: Supporting Activity Awareness in Collaborative UAV OperationsAwareness in Collaborative UAV Operations

• To develop and evaluate activity awareness tools for future teams of UAV operators engaged in time critical missions

• To investigate the software automation and interface requirements of UAV operations team members

- what type of information is useful to know about one’s team members and their task activities?

- what information do team members need to know about the UAVs’ activities?

- how does one’s job duties / team role impact activity awareness requirements (e.g. mission commander, UAV operator, etc.)?

Initial Research FocusInitial Research Focus

Future UAV Teams

Technological Support for the Team Supervisor

Assisting Assisting Team Supervisors Team Supervisors of UAV Teamsof UAV Teams

Research Approach:

1. Develop representative collaborative, time-critical C2 task scenario

2. Conduct cognitive task analysis (CTA) to generate system requirements for supervisory decision support and activity awareness displays

3. Develop supervisory displays in a simulated task environment

4. Conduct exploratory experiment

Representative Collaborative UAV Task Representative Collaborative UAV Task ScenarioScenario

Ground Force Protection:• Futuristic UAV Operations Team performing ISR in support of a

time-sensitive targeting for a ground convoy traversing a hostile region

Task Details:• UAV Operations Team:

- 1 Mission Commander

- 3 UAV operators, each controlling multiple semi-autonomous UAVs

• UAVs have camera sensors only

- Team must coordinate with external strike team

• UAV team can communicate with convoy & external intelligence sources

Mission Planning

Issues to be resolved in this phase:

Each UAV will have initial routes programmed

Each operator will have a pre-defined area under his/her responsibility

Helpful information for resolving these issues:

Time average target destruction will take based on -UAV speed :transit time

-engagement time: flagged EO imagery sent/operator confirm/communicate with strike team/strike team wait time)

AWACS, satellite and strike team schedule

Phase goals Phase Breakdown

ISR UAVOP monitors his region of coverage, while the UAVs search for

IED AWACS, Satellites, strike team and the convoy detect additional

enemies and communicate to the UAVOPs.

STR - Strike

UAVOP receives ISR information from the UAV and requests engagement authorization from the Mission Commander

The mCDR decides if the target should be stroke and either grants the authorization or denies it.

When granted, the UAVOP orders the strike team to engage the target The strike team coordinates with the UAVOP a time window for the

target engagement When the Strike team is ready to engage, the UAVOP places his UAV-

LE to pinpoint the target to be destroyed, during the appropriate time window

The strike tem engages the target of interest

BDA - battle damage

assessment

UAVOP performs Battle Damage Assessment, based on information provided by their UAVs, by the strike team and by the AWACS.

UAVOP analyzes the consistency to confirm the destruction of the target or to order a re-strike.

UAV-LEx down/

malfunction

All operators and mCDR are notified about the loss When the next target is found by the UAVOPx, the team is notified that

an UAV-LE is needed. Operators that can handover their UAV-LE send a signal (or a message)

and the mCDR decides whose UAV-LE will be handed over. If no other UAV-LE seems to be available, the mCDR decides whether

a UAV-LE should be handed over or not, according to their priorities and strategy.

UAV down/ malfuntion

All operators and mCDR are notified about the loss Operators that can handover a UAV send a signal (or a message) and the

mCDR decides which UAV will be handed over. If no other UAV seems to be available, the mCDR decides whether a

UAV should be handed over or not and if a re-zone should take place or not, according to their priorities and strategy.

Mission Execution

A Zone gets overloaded with

targets

The mCDR decides whether a re-zone should take place or not, according to their priorities and strategy.

If authorized by the mCDR, a re-zoned new map is available for all operators

All the routes should be reviewed The Situation Awareness Display is updated and an alert should notify

the changes

1 3

2

Hybrid Cognitive Task Analysis (Nehme et al., 2006):Step 1: Scenario Task OverviewStep 2: Event Flow Diagram

Step 3: Situation Awareness (SA) Requirements Step 4: Decision Ladders & Display Requirements

Generating Requirements for Assisting Generating Requirements for Assisting SupervisionSupervision

Activation: Threat situation was passed and convoy still holding (e.g. no known threat envelope for immediate future)

D3

IDENTIFY If threat level will remain low after convoy resumes mission path

Interpret: Decide whether to request convoy resumption

If holding back convoy, can release?If holding back convoy, can release?

Execute: Convoy resuming progress

Perceive convoy’s current and future threat level based on known targets’ locations/weapon’s range

with respect to convoy, current strike schedule, and uncertainty of surveillance

Perceive convoy’s current and future threat level based on known targets’ locations/weapon’s range

with respect to convoy, current strike schedule, and uncertainty of surveillance

MonitoringMonitoring

Send release request to convoy

Perceive convoy receives hold release

and is resuming mission path

Perceive convoy receives hold release

and is resuming mission path

RECOGNIZE if the convoy is released, threat level will remain low for immediate future

RECOGNIZE if the convoy is released, threat level will remain low for immediate future

Recognize if convoy communications link is active

Recognize if convoy communications link is active

Recognize if convoy communications link is active

Visual alert and possible audio alert of convoy holding; threat situation as shown by targets within threat envelope visual

Display measurement of threat level based on

threats within envelope

Display strength of convoy comm link

and availability

Display convoy acknowledgement if release received, and updated positioning of convoy

Regional map depicts threat locations and uncertainty of surveyed areas. Simultaneously depict information on known targets as well as up to date strike schedule.

Display Requirements4

Produces Functional & Information Requirements

Supervisory Displays & Simulated Task Supervisory Displays & Simulated Task EnvironmentEnvironment

Large-Screen Wall

Displays

Situation Map Display Mission Status Display

TabletPC Display

Mission Commander Display

Simulation & Collaboration Server

(Grouplab SharedDictionary)

Situation Map Display

Mission Status Display

Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness

• UAV status & tasking

• Current & expected convoy safety

• Current & expected operator task performance, relative to convoy safety

4

(nominal)

(down)

8

1

(reviewing ATR imagery)

Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness

Potential threat

envelopeTarget strike indicators

Known threat

envelope

Situation Map Display

Mission Status Display

• UAV status & tasking

• Current & expected convoy safety

• Current & expected operator task performance, relative to convoy safety

Situation Map Display

Mission Status Display

Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness

• UAV status & tasking

• Current & expected convoy safety

• Current & expected operator task performance, relative to convoy safety

Exploratory StudyExploratory Study

Subjects: 8 participants (including 6 NROTC)

Protocol:- 1 subject + 3 “confederates” per

experiment- 2 hrs training + 2 x 15 min trials- Retrospective interview

1 Subject as mCdr 3 “Confederates”

as UAV Operators

Situation Map Display

Mission Status Display

Some Preliminary ResultsSome Preliminary Results

• Convoy threat summary intuitive, useful for timing command decisions

- Particularly, visualizing relationship between strike team activity and convoy’s progress

• UAV status & tasking useful for deciding when to assist UAV operators

- However, lack of time-on-task cues required mental effort

• Operator performance displays rarely used, since surveillance progress was easily obtained from the Map

4 81

(nominal) (down)(reviewing ATR imagery)

Ongoing Work: Test-bed for Exploring Other Ongoing Work: Test-bed for Exploring Other Collaborative TST IssuesCollaborative TST Issues

UAV Operations Team 1. Interruption

recovery tools for the team supervisor

Ongoing Work: Test-bed for Exploring Other Ongoing Work: Test-bed for Exploring Other Collaborative TST IssuesCollaborative TST Issues

UAV Operations Team

2. Activity awareness interfaces for the UAV operators

3. New methods for deriving collaborative system requirements

1. Interruption recovery tools for the team supervisor

Expanded Simulated Task EnvironmentExpanded Simulated Task Environment

Large-Screen Wall Displays

Situation Map Display Mission Status Display

Mission Commander

Display (TabletPC)

Simulation & Collaboration Server

(GroupLab SharedDictionary)

Remote Assistance Display

Interruption Recovery Assistance for Team Interruption Recovery Assistance for Team SupervisorSupervisor

Mission Commander Display

(TabletPC)

Interactive event timeline, containing event “bookmarks”

Interruption Recovery Assistance for Team Interruption Recovery Assistance for Team Supervisor (cont’d)Supervisor (cont’d)

Clicking on an event bookmarkhighlights the location of the corresponding event on the map display, e.g., a convoy attack

Preliminary UAV Operator Display DesignsPreliminary UAV Operator Display Designs

Operator 1

UAV 4

Reroute/ReassignmentTarget ID Task

3UAV1

UAV 3 UAV 5

Health Status

Fuel level: 40%

Connected

- Searching - Searching Health Status

Fuel level: 70%

Health Status

Fuel level: 70%

Connected Connected 0:00:300:00:050:03:59 UAV3

5

Help

Request sent to:

UAV3

4 - Searching

Assistance Requests Overview

Req 3 (Complete)

Request from: Op 1 Elapsed time: 0m35sDetected by: UAV 3 Target type: 5 (Com Eq)

Request from: Op 3 Elapsed time: 0m30sDetected by: UAV 8 Target type: 5 (Com Eq)

Req 5 Request from: Op 1 Elapsed time: 0m05sDetected by: UAV 2 Target type: 3 (IED-V)

Req 4

ATR Information:

Possible Match: Target Type 5: Comm Equipment

Match Confidence: Medium

Found by UAV2 – Location: 43:00E/24:00N

ATR Decision:

SubmitClassification

Confirm ATR Suggested Classification:

Target Type 5 with Medium Confidence

Provide Alternate Target Classification:

Select Target Type…

Select Confidence Level…

Select Target Priority…

Veto Classification:

Image does not contain known target

Operator 3

Cancel

All

Others

Op 1

mCDR

Op 2

Classification:

Target Type 5 High Confidence

(no classification available)

Elapsed time:

0:00:20

Image Received at:

12:56:40

Request new image

Threat distance from the convoy: 15 miles

Elapsed Time:

Current Time:

1: 25: 30 PM

0: 22: 45

Mission Info:

0: 03: 28

UAV Reassignment

Elapsed Time:

Remaining Time:

0: 02: 37

Routes:

View Filter:

Hazards

Connection Blackouts

Targets Range

All

UAV 4

UAV 5

UAV 6

Sendprivate

All

Op 1

Op 3

mCDR

Strike team

others

Type here…

Op 2 – communication box

ALERTS & Clarification

SendType here…

Share Files

mCDR OthersStrike T Op 1 Op 3

Comm Status

0:03:57

On line

Handoff Available[0:00:30] Conference call

[0:01:20] mCDR to OP1:

[0:01:22] Op1 to Op3:

[0:01:25] Op3 to strike team:

[0:01:29] Op3 to Op2:

4 - Searching

Repair

Repair

RepairRepair

Health Status

Health Status

Health Status

Fuel level: 40%

Fuel level: 70%

Fuel level: 70%

- Target ID5

6

Connected

Connected

DisconnectedSafety route in 0:29:59

Reconnecting in 0:00:49

0:00:18

0:00:04

0:03:57

(0:01:50) UAV 7 (area3) has been shot down

(0:03:00) UAV 4 Fuel Level at 40%

(0:03:40) UAV 6 enter in a difficult area to connect

mCDR

Strike team

OP1

Others

OP3

0:03:57

0:03:57

0:01:53 - Changing shift

0:00:33 - Target task

Comm

Comm

Comm

Comm

0:03:57

Repair

Repair

Repair

Repair

Repair

Comm

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

MapDisplay

CommunicationsDisplay

Tasking Display:TargetID & rerouting, reassigning UAVs

Improved Methodology for Deriving Improved Methodology for Deriving Collaborative System Requirements Collaborative System Requirements

Project Goal:

• To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking

• Behavioral pattern detection and performance predictions

- Predictive algorithms (NN, HMM, Hybrid)

• Alert team supervising agent of sub-optimal cognitive strategies

- Stress & time pressure

- Rapidly evolving situation

Is Operator using “good” strategies?

Operator

Is the team doing well?

Supervisor

Excessive workload

Future Work: Team Performance PredictionFuture Work: Team Performance Prediction

• Behavioral pattern detection and performance predictions

- Predictive algorithms (NN, HMM, Hybrid)

• Alert team supervising agent of sub-optimal cognitive strategies

- Stress & time pressure

- Rapidly evolving situation

Is Operator using “good” strategies?

Operator

Is the team doing well?

Supervisor

Excessive workload

Future Work: Team Performance PredictionFuture Work: Team Performance Prediction

Team

ConclusionsConclusions

• Initial activity awareness designs show potential for facilitating supervisory-level decision making, and supporting teamwork in general

- Particularly, designs aimed at conveying current & expected status of interdependent activities

• Activity awareness approach (i.e. intelligent sharing of activity data) appears to be an appropriate design technique for large-screen displays in time-critical C2 environments

• Further work is needed to understand what activity information should be shared (and at what level of detail) during different UAV mission types and mission phases

Contact and Sponsor InformationContact and Sponsor Information

Contacts:

- Dr. Stacey Scott, sdscott@mit.edu

- Prof. Mary (Missy) Cummings, missyc@mit.edu

- Humans and Automation Lab (HAL) website: http://halab.mit.edu

Thanks to project sponsor:

- Boeing Phantom Works