developing performance predictive tools for supervisors of teams and complex systems

Developing Performance Predictive Tools for Supervisors of Teams and Complex SystemsFebruary 22nd 2007

Yves Boussemart ([email protected])Humans & Automation LabMIT Aeronautics and Astronauticshttp://halab.mit.edu

OutlineLab OverviewHuman Supervisory Control Single vs. Multiple UVsSupervising teams of UV operatorsTools for SupervisorTRACSPerformance PredictionTeam Environments

MIT Humans & Automation LabCreated in 2004Director: Dr. Mary (Missy) CummingsVisiting professors: Dr. Gilles CoppinVisiting scientist: Dr. Jill DruryPost Doctorate Associates: Dr. Stacey Scott, Dr. Jake Crandall, Dr. Mark AshdownGrad Students: Yves Boussemart, Sylvain Bruni, Amy Brzezinski, Hudson Graham, Jessica Marquez, Jim McGrew, Carl Nehme, Jordan Wan

Research and Current ProjectsResearch in the Humans and Automation Lab (HAL) focuses on the multifaceted interactions of human and computer decision-making in complex socio-technical systems.Time-Sensitive Operations for Distributed TeamsHuman Supervisory Control Issues of Multiple Unmanned Vehicles (Reduced manning)Measurement Technologies for Human-UV TeamsCollaborative Human Computer Decision MakingIntegrated Sensor Decision Support Sponsors: Office of Naval Research, Boeing, Lincoln Labs, AFOSR, Thales

Mary Cummings - Note change in funding agencies)

HAL Testing EquipmentSingle Operator Testing: ONRs Multi-modal Watch Station (MMWS)Team Testing: HAL Complex Operation CenterONR Mobile Testing Lab

Human Supervisory Control (HSC)DisplaysHuman Operator(Supervisor)

ActuatorsSensors

ControlsHuman Supervisory Control Humans on the loop vs. in the loop Supporting knowledge-based versus skill-based tasks Network-centric operations & cognitive saturation

Human-Supervisory Control of Automated SystemsProcess ControlUnmanned Vehicle OperationsSatellite OperationsManned Aviation(Mars rover)(Shadow UAV)

Major Research Area: HSC of Unmanned VehiclesShadow UAVPredator UAVPackbot UGVSpotter UGVUnmanned Aerial Vehicles (UAVs)Unmanned Ground Vehicles UGVs (i.e., Robots)Unmanned Undersea Vehicles (UUVs)VideoRay UUVOdyssey UUV

Mary Cummings - Make sure you make it clear we are not directly supporting all these vehicles!

Major Research Area: HSC of Unmanned VehiclesPackbot UGVSpotter UGVUnmanned Ground Vehicles UGVs (i.e., Robots)Unmanned Undersea Vehicles (UUVs)VideoRay UUVOdyssey UUVShadow UAVPredator UAVUnmanned Aerial Vehicles (UAVs)

Motivation: Increasing Reliance on UAVs in Military OperationsUAVs are becoming an essential part of modern military operationsTypical UAV missions include:Force protectionIntelligence, surveillance, and reconnaissance (ISR)Combat search and rescue Strike coordination and reconnaissance (SCAR)

Inverting the Operator/Vehicle RatioCurrent UAV Operations 1 UAV : 2-5 Operators

Semi-Autonomous UAV Operations 2-5 UAVs : 1 OperatorFuture UAV Teams

Developed large-screen supervisor displays that provide current and expected mission and task progress information of team assets and operator activityDisplays integrate related information and provides emergent features for time-critical dataCurrent Supervisory-Level Decision Support for Teams

Supervisory Information?Individual and Team performancesStress & time pressureRapidly evolving situation

Actions:Adaptive automationOperator replacement / shifts

Excessive workload

Towards Performance Prediction Tools4 step process:Tracking of individual actionsPattern recognition on strategies and performance predictionAggregation of individual data and collaboration factorsTeam level performance predictionsIs the Operator using good strategies?OperatorIs the team doing well?Supervisor

2-dimensional space: Level of Information Detail (LOID) Mode (action steps) 4 quadrants: LOID: higher vs. lower automation/information Mode: evaluation vs. generation of solutions

Technology disclosure for patent and licensingTracking Resource Allocation Cognitive Strategies (TRACS)

Example of TRACS Application Application: Decision-Support for Tomahawk Land Attack Missile (TLAM) Strike Planning Resource allocation task: Match resources (missiles) with objectives (missions) Respect Rules of Engagement Satisfy multivariate constraints Current system: PC-MDS, no decision-support3 interfaces at various levels of collaboration

Example of TRACS RepresentationTRACS applied to TLAM LOID: Higher automation: Group of criteria Individual criterion Lower automation: Group of matches Individual match Data cluster Data item

Mode: Evaluation: Evaluate, Backtrack Generation: Browse Search Select Filter Automatch

Example of TRACS ResultsMostly Manual(Interface 1)Mostly Automation(Interface 3)Combination(Interface 2)Cognitive strategies are emerging as patterns

Performance Prediction with TRACSTRACS as a observable data of Hidden Markov Model for individual users Compute the decision transition matrices from empirical data Bayesian Prediction based on Markov Chains

Performance Prediction with TRACS TRACS + Neural Networks: Detect pattern with neural network: cognitive strategies Alert supervisor when behavior degrades Are bad performances robustly predictable in advance?

Manual to automatchManual BrowsingAutomatch loop

UAV Team Operations

Collaboration FactorsIndividualTracking cognitive strategiesPerformance predictionsRelatively simple metrics TeamTeam dynamicsIntra-Team communicationVerbal & Non-VerbalPerformance metricsGroup AwarenessSituation and activity awarenessDistributed cognitionGroup Interaction Theories

Open Research Questions

Mary Cummings - you want to make it clear we could make predictions for human or automated supervisors

Critical Questions to ConsiderWhat metrics can we use to gauge team performance?Which factors drive the metric?

How does time pressure affect the decision process?

How much information does a supervisor need?Direct observation of operators behaviorSynthetic data only (TRACS)?Both?

SummaryFocus went from individual UAV operator to supervisor of teams of UAV operatorsProposing a performance predictive toolExtend the predictions to team environments

Questions?

Research supervised by Prof. M. L. Cummings Research effort sponsored by Boeing/Boeing Phantom Works Contacts: [email protected] [email protected] Web: http://halab.mit.edu TRACS demo: http://web.mit.edu/aeroastro/www/labs/halab/media.html http://tinyurl.com/ybafp2

Mary Cummings - combine this slide with the one before

Backup Slides

Interface 1 - manualLOA 2 - manual matchingBasic support: filtering, sorting, warning and summarizing

Interface 2 - collaborativeLOA 3 - collaborative matchingAdvanced features for interactive search of a solutionmanual matchingautomatch = customizable heuristic search algorithmgraphical summaries of constraint satisfactionoption to save for comparison purposes

Interface 3 - configuralLOA 4 - automated matching with configural displayHigh level, constrained solution searchno access to raw data

aggregated info. onlypossibility to tweak the solution or to force assignment

Tomahawk Mission PlanningPerformance on incomplete scenario

performance decreased when LOA increased on single interface setup best: interface 1 and interfaces 2&3 - worst: interfaces 1&3 no deviation on interface 3

Interface 1: P = 69.7 Interface 3: P = 68.5

Problems with a 3D visualization Loss of granularity and clutter Occlusion effect (loss of 2D information) Parallax effect (detrimental perspective) Difficult to manipulate (high cognitive load) Difficult to orient oneself (loss of SA) Lack of emergent temporal analysis featureTRACS 3D

From TRACS 3D to TRACS 2.5DTemporal Data TRACS 3D: orthogonal axis TRACS 2.5D: interactive timeline

Advantages Not 3D (occlusion, parallax, orientation problems addressed) Familiar manipulation Clear grouping of temporal features (granularity, clutter, emergent properties)

TRACS 2.5D

Humans and Automation LaboratoryMobile Advanced Command and Control Station

Mobile Advanced Command and Control Station

This is what we did so far, display onlyNow, next step algorithmic helpPersonal Computer Mission Distribution System Tomahawk Command Issue of noiseHow much in advance?What constitute a the right strategy?Metrics?GA: predator Ground Control StationNon trivial collaboration factorsMITRE added in as sponsored? I mean, I know I didnt do that much :-/

developing performance predictive tools for supervisors of teams and complex systems

Documents