2009, aptima, inc. 1 ma ▪ dc ▪ oh ▪ fl © 2009, aptima, inc. human-centered engineering...

2009, Aptima, Inc. 1

www.aptima.comMA ▪ DC ▪ OH ▪ FL

© 2009, Aptima, Inc.

Human-Centered Engineering Perspectives on Simulation-Based Training

Daniel SerfatyEmily Wiese

Presented to SimTransCopenhagen, Denmark22 June 2009


Agenda

Introduction to Aptima– Examples of Capabilities in Human-Centered Engineering

Five Emerging Technologies in Simulation-Based Training– Scenario Engineering– Simulation Fidelity– Performance Measurement (with A-Measure toolkit demo)– Cognitive Skills Training– Team Communications Assessment

Discussion


What is Human–Centered Engineering?

TechnologyCapabilities

Social &Organizational

Structures

Mission, Tasks & Work

Processes

Human Agents

Congruence


Aptima, Inc.

Interdisciplinary Small Business – Founded in 1995– Consistent annual growth (40% CAGR)– 100+ staff (80% graduate degrees)

Human Centered Engineering– Analyze and design complex socio-technical

systems– Combine social science theory with

quantitative, computational methods Serving government and commercial clients

– 350+ contracts with the Defense Industry Offices

– Boston/Woburn, MA, (HQ) – Washington, DC – Dayton, OH– Ft Walton Beach, FL


Domain Expertise

Skill Set

Command & Control Military Training Leadership Complex Information Display National Security Solutions Medical & Healthcare Aviation Emergency Preparedness Stability & Support Operations Education Safety

Educational Backgrounds


Optimizing Performance in Mission-Critical Environments


Examples of Capabilities

Performance Measurement Socio-Cultural Applications Organizational Engineering Training


Official U.S. Navy photo. Neither the U.S. Navy nor any other component nor any other component of the Department of Defense has approved, endorsed, or authorized this product [or promotion, or service, or activity].


Agenda

Introduction to Aptima– Examples of Capabilities in Human-Centered Engineering

Five Emerging Technologies in Simulation-Based Training– Scenario Engineering– Simulation Fidelity– Performance Measurement (with A-Measure toolkit demo)– Cognitive Skills Training– Team Communications Assessment

Discussion

The New Science of Scenario Engineering

BEST– Engineering the Stimulus for Optimal Learning

PRESTO– Optimizing Learning Trajectories Using Constraint-Based Logic

CROSSTAFF– Engineering Training Scenarios from Operational Data

VSG– DDD Visual Scenario Generator Tool

Q: How to Optimize Learning on a Given Simulator?

Vision: Learning from Synthetic Experiences

Training Opportuniti

es

Training Experienc

esNext Scenario

Readiness

Competencies, Knowledge, Skills…Training

Objectives

Phase I

0

4

8

12

16

20

flag alpha bravo charlie

Cell

Worklo

ad

selfrating

othersrating

Phase II

0

4

8

12

16

20

flag alpha bravo charlie

Cell

Worklo

ad

selfrating

othersrating

What to Measure

Performance Measurement

Data

BEST: Optimizing Scenario Selection

Assess team performance against near-optimal solution Based on that assessment, select training events to

optimize the team’s learning curve

R2 = 0.966 R2 = 0.9841

41000

42000

43000

44000

45000

46000

47000

48000

Mission

Te

am

De

fen

siv

e S

co

re

MostControlMost TrendlineControl Trendline

Training Session

Tea

m S

core

What sequence of experiences moves a team to a steeper learning curve?

System’s Belief about State of Competencies

True State of Competencies

s(n-1)s(n-1) s(n)s(n) s(n+1)s(n+1)

X(n-1)X(n-1) X(n)X(n) X(n+1)X(n+1)

scenarioat n

scenarioat n

rewardat n

rewardat n

observationat n

observationat n

scenarioat n+1

scenarioat n+1

rewardat n+1

rewardat n+1

Parameters set by SMEs

Parameters set by SMEs

Tool: Partially Observable Markov Decision Process (POMDP) Training Model

Experiment Result:52% Improvement over Best

Practices Team Training

Conventional vs. PRESTO-Based Scenarios

Conventional Scenario MSEL EventsPlanned Actually Occurred

Didn’t Occur

PRESTO-based ScenarioActual

Actual

PRESTO-Based Scenario Space

(a) Original Flight Path (b) Key Events Identified

(d) Scenario Envelope Generated(c) Events Generalized

Engineering Training Scenarios from Operational Data (CROSSTAFF)

Scenario Engineering Using the DDD* Visual Scenario Generator (VSG)

DDD*: Distributed Dynamic Decision-making simulator

Pedagogically-focusedAdaptive Scenarios

Scenario

Initial

Conditions

Participant

Advice

Exercise

Events

Training Objectives

TO Conditions

Ongoing TrainingExercise

PRESTO/CROSSTAFF

SAF/Instructor

Advice

Performance Measurement

System

(A-Measure)

Trainee Performance

Measures

BEST Optimal Vignette

Sequencing

Understanding Simulator Fidelity Requirements

1. There is little guidance and no standard tool for determining the appropriate level of fidelity of training simulators to – Achieve specified training

objectives,– Maintain trainee acceptance,

and – Fit within budgetary constraints.

Simulator Fidelity

Tra

inin

g E

ffect

ive

nes

s

2. There are no standard measures designed to be sensitive enough to detect objective performance differences invoked by varying levels of fidelity

Perceived

Act

ual

Co

st

2009, Aptima, Inc.

31

RELATE: A Research-Driven Approach

A systematic approach to establish quantitative, predictive relationships between simulator fidelity and training effectiveness

RELATE fuses:– Fidelity requirements defined by end-users;– Existing theory and research about fidelity; and – Objective performance data from fidelity experiments to develop a

predictive, computational model.

2009, Aptima, Inc.


Model-Based Tool

A computational model-based tool to assist with decisions regarding the acquisition and use of training simulators

Tool can help users: Conduct return on investment

analyses to determine which simulator to develop or acquire

Prioritize technology enhancements to improve the effectiveness of existing simulators

Develop an strategy for employing both high- and low-fidelity simulators to meet training objectives


PERFORM: Air-Air Combat Fidelity Requirements

Conducted research to examine the visual and cockpit fidelity requirements for training air-to-air combat skills to experienced pilots in F-16 simulators

Developed a decision-support tool to help the Air Force prioritize technology enhancements to improve the effectiveness of deployable simulators

Deployable Tactics Trainer (DTT)Display for Advanced Research and Technology (DART)

34

PREDICT: Air-Ground CombatFidelity Requirements

Conducting research to examine the visual fidelity requirements for training air-to-ground combat skills to inexperienced pilots in F-18 simulators

Developing a decision-support tool to help the Navy develop a strategy for employing both high- and low-fidelity simulators to meet training objectives

2009, Aptima, Inc.

35

FLEET: Developing and ValidatingFidelity-Sensitive Measures

Developing measures of pilot performance that are sensitive enough to detect objective performance differences invoked by varying levels of fidelity

2009, Aptima, Inc.

Fidelity-Sensitive Performance Measures

Does the pilot complete Combat Fence IN Checks?

Does the pilot maintain briefed formation?

Does the pilot appropriately mitigate surface-to-air threats?

Does the pilot deconflict with other assets?

Conducting research to examine the motion fidelity requirements for training air-to-ground combat skills to inexperienced pilots in T-45 simulators

Summary

Aptima seeks to identify the simulator fidelity requirements for effective training by: Developing a systematic approach to establish quantitative,

predictive relationships between simulator fidelity and training effectiveness

Building a computational model-based tool to predict the impact of simulator fidelity on training effectiveness

Creating performance measures and measurement tools that can be used to collect better data in simulator fidelity experiments

Employing the proper level of fidelity will ensure better training results and reduce costs by eliminating investments in unnecessary training and technology

2009, Aptima, Inc.

New Approaches to Measuring Performance

A well-designed measurement system makes simulation-based practice effective training– The right feedback to the right person at the right time leads to better

learning

Measurement enables assessment of training effectiveness – Are people getting the skills they need?

Guides selection of training environment– Live, virtual, constructive

– Facilitates appropriate use of measures

Measurement technology can turn simulators into training machines

You Can’t Train What You Can’t Measure

Why is this hard?– Volume of data

– “We recorded everything...”

– Lack of meaningful aggregation methods– “325,435 messages were received…the average length was 2.35 minutes”

– Interdependence of behaviors at different locations– No one person has the total picture

– “Correct” behavior depends on dynamic context– It is hard to construct, even after the fact, where the team went wrong

Hours spent training ≠ Proficiency

Real-world experience ≠ Proficiency

Performance Measurement Process

Competency Based Performance Measurement

Competency-based performance measures

Leverages performance measurement theory in combination with subject matter expert input

Assesses team and individual performance

The COMPASSSM

Methodology is a product of Aptima, Inc.

COMPASSSM tells us the what direction to go with measure development

A▪Measure Product Family

Turning simulators into training machines

Cognitive Skills Training

Not all skills should be trained the same way– Cognitive skills vs. procedural skills

Pedagogical Theory: Direct instruction vs. constructivism Most types of training/education employ a direct instruction

approach– Can be effective for training procedural skills

Current research suggests that a problem-based approach may be more appropriate for training cognitive skills– What differs is when you tell someone how to do something relative to

when they practice doing it E.g., telling students the formula for density and having them practice (the traditional

approach) enforces a plug-n-play understanding. Very little transfer. Instead, give students a situation and ask them how they would describe density.

– Now they get a sense of the principles involved before you give the formula.

The Bransford Model

Community-Centered(Relevance)

Assessment-Centered(Formative Feedback)

Knowledge-Centered(Deep Understanding)

Learner-Centered(Where do Learners Start)

Aptima’s Balanced Unified Incremental Learning Development (BUILD)

Training Approach

Example: How to “train the ear”

Air Battle Managers (ABMs) monitor multiple communications (radio) channels at the same time

This currently an acquired skill developed through experience and on the job training.

How can we train a novice?

At its core, this is a skill that relies heavily on cognitive skills like memory and attention.

Monitoring multiple communications channels requires…– Dedicating limited cognitive resources (memory)

– To attending to stimuli that must pass a certain threshold (attention)

– Under stressful conditions (stress)

How can these cognitive concepts help ABMs?

“Monitoring multiple comms channels requires…– Dedicating limited cognitive resources (memory)”

Psychological research demonstrates that one can free limited working memory resources by placing some information in long term memory.

The key to long term memory storage is automaticity.– When I see X, I do Y

The key to training automaticity is repetition– Multiple trials

How can these cognitive concepts help ABMs?

“Monitoring multiple comms channels requires…– Dedicating limited cognitive resources (memory)”

– To attending to stimuli that must pass a certain threshold (attention)”

The threshold is often physical (a certain volume, a certain brightness), but could also be semantic (meaningful).– Cocktail party effect

Trigger words can break the semantic threshold for attention– When I hear an important word, I attend to it

This is the mechanism for retrieval.

From Theory to Application: A Layered Comms Training Approach

Phase I: Trainees are introduced to ABM trigger words and their definitions. – Begin to store in long term memory

Phase II: Trainees recognize trigger words within a stream of communications.– Develop automaticity and retrieval

Phase III: Trainees recognize a trigger word in a realistic scenario and respond accordingly.– Automatically retrieve under stressful conditions

Cognitive Skills Training

CIFTS: Communications Analysis in Operational Environments


Domain: Air and Space Operations Center (AOC)

Numerous centers around the world

Around 100 operators communicating In the air On the ground Around the world

Extremely complex operations must be coordinated

Extensive use of chat to coordinate, assign tasks, exchange information

“Communications is at the Heart of Team Performance”

CIFTS Project Domain: AOC Chat Data


[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> firing unit how can rtb green up[23:09] <SOLE> tdn shot down fac[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position standby dtl updated[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TDO> TGTS: please might mean[23:12] <TGTS> sir bent no[23:13] <SOLE> high fast flyer affirm[23:13] <TGTS> c

[23:09] <GTC> still orthanc where unsuccessful[23:09] <TDO> firing unit how can rtb green up[23:09] <SOLE> tdn shot down fac[23:09] <TDO> GTC: relay inform pol leadership lost[23:10] <GTC> current position standby

[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> firing unit how can rtb green up[23:09] <SOLE> tdn shot down fac[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position standby dtl updated[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TDO> TGTS: please might mean[23:12] <TGTS> sir bent no[23:13] <SOLE> high fast flyer affirm[23:13] <TGTS> c[23:14] <CCO> Chief: attack check jdocs sado right wrkg unfriendly lost track[23:14] <GTC> SOLE: anyone did you copy can you confirm[23:15] <SADO> ATK: what up[23:16] <GTC> still issues[23:16] <TGTS> TDO: wrking wrong sam ring return to base facilities who is disgard[23:16] <SADO> SODO: jadocs each time wrong voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <GTC> jstars armored car link track[23:17] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> resend[23:21] <GTC> not good wmd vehicles where[23:22] <SOLE> wmd vehicles ukn roll call requested do u agree[23:23] <SADO> GTC: all c2 players do u launch pads rqist control measure was back up[23:23] <CCO> no idea[23:25] <Chief> SADO: strike asset changing cco[23:25] <SODO> negs cvy need no pred[23:25] <SOLE> what were you doesnt vehicle assembly msn results[23:25] <SIDO> ATK: got it mass of vehicles pimp

Need automated methods to understand what is happening in general, and the ability to drill down to specific instances

Advanced Language Analysis:LAVA TOOLkit

LAVA is Aptima’s LAtent Variable Analysis toolkit LAVA provides tools for natural language processing

– Processing free text to represent words as numbers– Statistical tools to extract concepts from free text

LAVA is language-independent– Inter-agency, inter-cultural, international differences

The kinds of questions you can ask of LAVA– What are the main concepts within this set of news articles?– How similar are these two medical records?– What is this e-mail talking about?– How does the subject change over time?

LAVA in JAVA– Intel Math Kernel Library– Microsoft SQL Server or MySQL– Web services– JAVA API

52 2009, Aptima, Inc.

PACE: Processing and Analysis of Communications and Events

Addressee Density

– High Density Message Type

– Question– Acknowledgment– Command– Ambiguity– Pause

Valence– Positive– Negative

Mission ID– JA0002 == Underground Bunker

Chains– GTC – TDO

Combinations….


[23:09] <GTC> TDO: still orthanc where unsuccessful[23:09] <TDO> SOLE: can we green up?[23:09] <SOLE> c, fac shot down[23:09] <TDO> SOLE: relay to inform pol leadership lost[23:10] <GTC> current position JA0002 standby dtl[23:11] <Chief> TGTS: thinks minus one chem fac why[23:12] <TDO> TGTS: hvaa retrograde transport type[23:12] <TGTS> what does hvaa mean?[23:12] <TGTS> sir bent no[23:13] <TGTS> please confirm high fast flyer[23:13] <TDO> c[23:13] <TGTS> Chief: attack JA0002 check jdocs[23:14] <GTC> TDO: did you copy?[23:15] <TDO> sb[23:16] <ATK> still issues[23:16] <TGTS> TDO: wrking return to base JAoo2[23:16] <SADO> SODO: jadocs each time voice comms[23:17] <ATK> kill call underground bunker correct atk[23:17] <SODO> jstars armored car link track[23:18] <TGTS> GTC: facility oga rotary wrkg[23:18] <GTC> cpy[23:21] <TDO> GTC: where are wmd vehicles?[23:22] <SOLE> wmd vehicles ukn








Non-Combatant Evacuation Data Example: Valence and Type


CIFTS Timeline Interface


Tick marks indicate messages in timeline. Colors indicate

values for the current analysis.

Any two analyses can be “crossed”

Messages can be filtered by

checking values on each analysis type

Tabs display different analyses

Content of messages

Tabs hold different kinds of summary data

CIFTS Network Interface


Arrow A->B indicates message from A was followed by a message from B within the same chat room; on average, a “response”

Arrow thickness indicates % of messages from A that were followed by messages from B; “conversations”

Size of the circle indicates percent of all messages that this member sent

Length of the line indicates the time interval between messages; “density” of comms

Layout tries to minimize line-crossings; localizes “functionality”

More “central” members have contacts with more other members

Conclusions & Questions

Key Questions:– In a world dominated by simulation-based training,

how do we optimize learning yield (ROI)?– In scenario-based training, what is the “curriculum”?– How can we get effective and efficient in performance

measures selection and feedback?– How can rigorous scientific methods help contribute to

the above?– Scientific methods Practical software tools?

Daniel Serfaty: [email protected]

Emily Wiese: [email protected]

2009, aptima, inc. 1 ma ▪ dc ▪ oh ▪ fl © 2009, aptima, inc. human-centered engineering...

Documents