Performance Assessment in Simulation-Based Team Training: Principles from the Science That Matter

Marissa L. ShufflerEduardo Salas

University of Central FloridaFebruary 11, 2010

Performance Testing Council Summit, Orlando FL

Overview What do we know about teams and team

performance assessment? What is simulation-based team training (SBT)? How do we assess performance in team SBT? Best practices for SBT performance assessment

• Two or more individuals

• Multiple information sources

• Meaningful task interdependencies

• Coordination among members

• Common, valued goals

• Intensive communication

• Adaptive mechanisms

• Hierarchical structure

• Specialized member roles and responsibilities

• Task-relevant knowledge

Defining Characteristics of Teams

Driving Question:

How Do We Turn a Team of

Experts into an Expert Team?

The Conditions and Processes of Team Performance

Team Performance Assessment Systems

Requirements of a team performance measurement system (Cannon-Bowers & Salas, 1997): Must be theoretically based

Models of team performance Cognition, behavior and affective factors

Must consider multiple levels of measurement Individual/Team Process/Outcome

Must be able to describe, evaluate and diagnose performance Moment to moment changes Determine causes Help with remediation

Must support ease of use Easily adaptable Useability

Must have validity & good psychometrics Face External/Internal

Must facilitate Debriefing/AAR Feedback

Team Performance Assessment Systems

What We Know about Team Performance Assessment We have a good theoretical basis…

138 models! (Salas, et. al. in press) We have an idea (not perfect) of what to measure…

Team Type Taxonomies (Sundstrom et al, 2000) Team Competency Taxonomies (Cannon-Bowers et al,

1995) Shared Mental Models (Cannon-Bowers et al, 1993)

There are some tools available that work (not perfect)… Qualitative/Quantitative Outcome/Process Taskwork/Teamwork Used in Complex Simulations

What is Simulation? “Realistic synthetic work/learning environments

that can be used for training” (Andrews, 2005) What is Simulation-Based Training?

“A training strategy that uses several aspects of scientific, theory-based training, incorporating information, demonstration, and practice-based methods” (Salas, Priest, Wilson, & Burke, 2006)

Defining Simulation-Based Team Training

Defining Simulation-Based Team Training

What is Team Training? “A set of tools and methods that, in combination

with required competencies and training objectives, form an instructional strategy” (Salas & Cannon-Bowers, 2000)

Defining Simulation-Based Team Training

Used to accelerate expertise in team members…

Explicit links are developed and maintained among training requirements, exercise design, observation, and feedback…

Defining Simulation-Based Team Training

SBT provides trainees opportunities to demonstrate mastery of learned teamwork skills…

SBT provides opportunities to turn a team of experts into an expert team…

SBT Works Best With… …Clear & Precise Learning Outcomes …Carefully Crafted Scenarios …Well Designed Performance Measures …Robust Observation Protocols

Why Measure Performance in Team SBT? Quality measurement practices guide learning

and corrective feedback Measurement can also ensure learners possess

the requisite competencies for effective team performance

Practice opportunities in SBT can be used to detect deficiencies in both individual and team competencies

Needed Links in Simulation-Based Team Training

PerformanceHistory

Feedback

MeasuresScenarios &

EventsLearningOutcome

Tasks & KSAs

Challenges to Assessing Performance in Team SBT Measuring teamwork is complex!

Multiple people and multiple skills means multiple measures are required

Measuring the critical aspects of performance instead of measuring what is easy A single snapshot of performance may not be completely

representative of long term performance

It is difficult to remove observers or raters from the assessment process Raters are not always reliable!

How to Measure Team Performance in SBT Two important decisions from a training

perspective to guide assessment development: What meaningful feedback should be given to the

team and each individual member? What further training is required by team or

individual members? Depending on the answers to these questions,

certain methods may be more or less useful

Methods of Team SBT Assessment Event based measurement

Generates behavioral checklists that are linked to scenario events and KSAs being trained

Behaviorally Anchored Rating Scales (BARS) Provides brief descriptions of behaviors as

anchors associated with each particular rating

Methods of Team SBT Assessment Behavioral Observation Scales (BOS)

Generally uses a Likert type scale to rate the frequency of certain team processes

Self-Report Measures Questionnaires administered to each team

member individually

Best Practices for Team SBT Performance Assessment

Drawing from what we know regarding SBT and team performance assessment, several guidelines have emerged

These guidelines are theoretically, empirically, and practically based Derived from literature reviews, observations,

and interviews (Salas, Rosen, Weaver, Held, & Weissmuller, 2009)

Best Practices for Team SBT Performance Assessment

Best Practice 1: Ground measures in theory Use theory to answer questions about what to

measure Capture aspects of Input Process Output

models of team performance

Best Practice 2: Design measures to meet specific learning outcomes Clearly articulate the purpose of measurement at

the beginning of the development process Design measurement systems to capture

information necessary for making decisions about learning outcomes

Best Practices for Team SBT Performance Assessment

Best Practice 3: Capture competencies Explicitly link performance measures to

individual and team level competencies targeted for training

Best Practice 4: Measure multiple levels of performance Performance measures should be sensitive to

differences in individual and team performance

Best Practices for Team SBT Performance Assessment

Best Practice 5: Link measures to scenario events Link performance measures to opportunities to perform in

the simulation Affords greater structure & standardization for observers

Best Practice 6: Focus on observable behaviors Decreases “drift” of observers ratings over time Provides a fine level of granularity that can help in

providing feedback

Best Practices for Team SBT Performance Assessment

Best Practice 7: Incorporate multiple measures from different sources Different sources provide unique information Must develop a plan for integrating information

Best Practice 8: Capture performance processes in addition to outcomes Should provide information about HOW teams reached

their outcomes so that poor process can be corrected

Best Practices for Team SBT Performance Assessment

Best Practice 9: Create diagnostic power Provide information about the causes of effective and

ineffective performance

Best Practice 10: Train observers and structure observation protocols Establish training to ensure observers are accurately and

reliably rating performance Provide structured protocols to ease the information

burden of observation

Best Practices for Team SBT Performance Assessment

Concluding Remarks Team performance assessment is not easy, but

there is a wealth of research and theory that exists to support assessment tool development…so…

Use the science!

Thank You!

Questions?