a psychological fidelity approach to simulation-based training - iacmr

A Psychological Fidelity Approach to Simulation-Based Training: Theory, Research, and Principles Steve W. J. Kozlowski and Richard P. DeShon Michigan State University

Draft 1: 20 July 2000 Final Submission: 20 September 2002

Kozlowski, S. W. J. & DeShon, R. P. (2004). A psychological fidelity approach to simulation-based training: Theory, research, and principles. In E. Salas, L. R. Elliott, S. G. Schflett, & M. D. Coovert (Eds.), Scaled Worlds: Development, validation, and applications (pp. 75-99). Burlington, VT: Ashgate Publishing.

Psychological Fidelity 2 Abstract Transfer in terms of skill maintenance and generalization is one of the key challenges in effective training design. Training design for critical military tasks often entails the use of simulations that are high on physical fidelity to minimize skill decrements that impede training transfer. Although this is an effective approach to resolving the transfer problem, it can be costly and inefficient. Moreover, a new wave of advanced training technologies based on a distributed network architecture is emerging. These emerging training technologies will rely on low fidelity synthetic tasks to address the transfer problem. This chapter presents a theoretically based strategy for training research and design that focuses on psychological fidelity--an explicit effort to model underlying psychological constructs and processes responsible for effective performance. Theoretically driven psychological fidelity is essential for enhancing the training to transfer linkage. We argue that a focus on psychological fidelity can leverage the training potential of cost-efficient low fidelity simulations used in distributed training systems, and can enhance the effectiveness of high fidelity simulations as well. The chapter presents the principles of this approach, illustrates their research application, and describes the advantages of explicitly considering psychological fidelity in training research and design.

Psychological Fidelity 3

A Psychological Fidelity Approach to Simulation-Based Training: Theory, Research, and Principles

Steve W. J. Kozlowski and Richard P. DeShon Michigan State University

Many critical activities, such as air traffic control, industrial process control, and military

command and control, are accomplished by individuals and teams interacting through complex,

technology mediated systems. These task environments, which can be characterized as dynamic

decision making (DDM) situations, place high demands on the skills and capabilities of operators.

DDM tasks are dynamic, ambiguous, and emergent, necessitating rapid assessment of the situation as it

unfolds, diagnosis and prioritization of possible actions, and implementation of appropriate task

strategies. DDM tasks place heavy demands on decision makers, necessitating high levels of expertise

to enable the strategic action and adaptive performance required for team effectiveness.

How can training be used to develop the knowledge and skills essential to strategic and adaptive

performance? From a training design perspective, there are two critical and related issues that need to

be addressed: skill acquisition and transfer (Goldstein, 1993). Skill acquisition concerns learning the

knowledge and skills necessary for effective performance. Transfer concerns the transportability of

trained knowledge and skills from the training context to the performance environment, and focuses on

issues of retention, maintenance, and generalization (Baldwin & Ford, 1988). Although both issues are

important, transfer is the more challenging issue (Barnett & Ceci, 2002). Moreover, transfer is directly

relevant to adaptabilityBan essential aspect of team effectiveness in DDM environments.

There are two general approaches for resolving the transfer problem through training design.

One approach addresses physical fidelity, whereas the other approach addresses psychological fidelity.

The physical fidelity approach focuses on the use of high fidelity simulation during the skill acquisition

phase in an effort to minimize or eliminate skill degradation during transfer. High fidelity simulation is

an effort to design a training context that physically reproduces the actual performance environment to

Psychological Fidelity 4 the greatest possible extent. The equipment to be used; its controls, reactions, and behavior; its look,

feel, and motion are made to be as realistic as possible. Training or practice scenarios are often based

on actual events to further enhance the realism of the training experience. The essence of this training

strategy is that the emphasis on realism will minimize differences between the training and performance

contexts, thus enhancing the potential for knowledge and skill transfer. It is a theoretically based (i.e.,

identical elements theory: Thorndike & Woodworth, 1901), tested, and effective approach (Druckman

& Bjork, 1991) to the transfer issue. It is the dominant approach to training design for critical military

tasks such as aviation crews and command and control teams. However, it is also a training strategy

that can be costly, time consuming, and inefficient.

The other approach -- one that we incorporate in our research -- focuses on psychological

fidelity in training design. Psychological fidelity concerns the extent to which the training environment

prompts the essential underlying psychological processes relevant to key performance characteristics in

the real-world setting. In other words, it is an effort to evoke the central psychological constructs and

mechanisms responsible for on-the-job performance. Whereas the physical fidelity approach attempts to

accomplish this implicitly by replicating the performance environment, the psychological fidelity

approach represents an effort to model this explicitly by using basic theory to guide research and

training design. By doing so, it has the potential to enable the use of cost-effective low fidelity

simulations during training that can nonetheless maximize transfer in terms of retention and, more

importantly, generalization.

It is important to recognize that these two approaches are not competing alternatives; rather,

they are complementary. It is simply that the physical fidelity approach dominates training design for

transfer, and the psychological fidelity approach is an emerging perspective. One can consider a

training system to be a series of episodes or experiences that systematically build key skills from basic

to strategic to more complex adaptive skills (Kozlowski, 1998). Low fidelity simulation has an

Psychological Fidelity 5 important role to play in such a system. Moreover, a new wave of distributed training technologies is

emerging that will make extensive use of low fidelity synthetic tasks. These technologies will be based

on networked architectures linking distributed PC platforms that can run complex interactive

simulations. Individuals, teams, and teams of teams will linked together to engage in common training

experiences. The cost advantage and training potential of these emerging training systems is

inescapable. However, the technologies are merely media for delivering information and experience;

they are not instruction per se. To be effective training systems, they will have to meet the challenges

of skill acquisition and transfer. Thus, a critical question concerns how these challenges can be met

through the use of low fidelity simulations. We assert that psychological fidelity is an essential feature

of training design regardless of the level of physical fidelity of the simulation. Indeed, when coupled

with the physical fidelity approach, the psychological fidelity approach can improve the cost-benefit and

overall effectiveness of the training system.

Our purpose in this chapter is to explicate the principles of psychological fidelity and to

illustrate how we utilize these principles to guide our basic research on training design for skill

acquisition and generalization. The hallmark of our approach to enhancing psychological fidelity is the

centrality of theory; it is essential at each phase of the research process. We begin with a brief

discussion of adaptability and provide an overview of the theoretical heuristics that guide our research

program on the development of adaptive performance skills. We next define and describe the essential

elements of psychological fidelity, and the research strategies that translate the elements into

experimental designs. We provide an illustration of this approach through work in our research program

on training and developing adaptive performance. And, finally, we close with a brief discussion

regarding the advantages of our approach as a research strategy, and as a method for developing tools

for simulation-based training.


Problem Background and Theoretical Overview

What is Adaptability, Why is it Important, and How is it Developed?

Dynamic problem situations create challenges for decision makers and place a premium on the

capability to adapt individual and team performance to the shifting demands of the emerging problem

situation (Orasanu & Connolly, 1993). The problem is ill structured, with incompatible or shifting

goals. Diagnostic information is difficult to obtain, and is often ambiguous or conflicting when it is

available. The situation is dynamic and emergent, responsive to decision maker actions, but also subject

to unpredictable shifts. Individual decision makers are embedded in teams, and must coordinate their

individual efforts with multiple players. Often there are significant time pressures and high stress.

Thus, DDM situations call for more than the static and routine application of well-learned

knowledge. Such situations necessitate what Holyoak (1991) describes as Aadaptive expertise,@ and what

we refer to as adaptability or adaptive performance. Adaptive performance builds on a foundation of

basic domain knowledge and the routine expertise that guides performance in typical situations.

However, adaptive performance goes beyond procedural knowledge of an automatic sort. It requires

active cognitive monitoring to develop a deep comprehension of the conceptual structure of the problem

domain. Adaptive experts understand when and why particular procedures are appropriate, and also

when they are not. Comprehension entails mindful processing, allowing adaptive experts to recognize

shifts in the situation that necessitate adaptability (Smith, Ford, & Kozlowski, 1997).

A key factor for the development of adaptive performance skills is active learning during skill

acquisition. Active learning enhances the development of metacognitive and self-regulatory skills.

Metacognition refers to executive-level processes entailing knowledge, awareness, and control of

cognitive activity involved in goal attainment (Flavell, 1979). Self-regulation occurs at a more micro-

level, and entails the planning, monitoring, and adjustment of cognitive and task strategies necessary to

accomplish sub-goals. In addition to cognitive and task-relevant strategies, self-regulatory skills entail

Psychological Fidelity 7 the capability to manage affect. Complex tasks require focused attention and cognitive effort. Tasks that

are difficult mean many errors and frustrations early in the learning process. The negative affect that

accompanies failure to meet expectations draws attention away from the task and must be managed.

Effective management of the learning process enhances self-efficacy, a sense of self-perceived task

competency that allows the individual to tackle difficult tasks and persist in the face of novel challenges.

These capabilities are also important for maintaining motivation under challenging and shifting

performance conditions (Bandura, 1991; Bandura & Wood, 1989).

For teams, metacognitive and regulatory processes must extend beyond the self. That is, these

individual-level cognitive and behavioral skills must operate in a coherent fashion across the team.

Individuals must maintain an awareness of self within the network of roles that comprise the team.

They must monitor the rhythm, timing, and pacing of team activity to enable coordination. They must

monitor the performance of critical interdependent roles, and be prepared to step in and share the

workload when teammates become overloaded. They must build and maintain a sense of team efficacy

to deal with challenges. And they must be capable of revising tasks, roles, strategies, and goals across

the entire team when the situation demands adaptation on the fly (Kozlowski, Gully, Nason, & Smith,

1999).

Clearly, adaptive performance skills are critical to the effectiveness of individuals and teams

operating in DDM environments. Active learning needs to be stimulated during skill acquisition to

enable individuals and teams to generalize under transfer. What theories of learning, training, and

development can guide this process?

Theoretical Foundation

A solid theoretical foundation is central to our approach of building psychological fidelity into

our research program. Three theoretical legs support this effort. The first leg is formed by basic theory

pertaining to fundamental psychological processes involved in learning, motivation, and performance --

Psychological Fidelity 8 theories of action initiation and self-regulation of cognition, behavior, and affect. This leg is at the core

of our research effort. The second leg is formed by theories of instructional design, particularly those

that address the development of individual-level adaptability and skill generalization. This leg allows us

to identify instructional interventions with the potential to influence core psychological constructs and

processes. The third leg is formed by broader meta-theories or heuristics that address the training and

development of adaptive teams, and multilevel issues. This leg provides a framework for determining

how individual-level learning and performance translate into team level processes and outcomes.

Together, these three theoretical legs provide an integrated foundation for our research addressing the

development of teams with adaptive performance skills. We briefly describe each of the theoretical

perspectives below.

Self-regulation. The dominant paradigm in current research on the initiation and control of

action is termed self-regulation theory. Self-regulation theory has developed a broad base of empirical

support as an effective model of the cognitive, behavioral, and affective mechanisms that contribute to

learning and task performance. Although there are several different models of self-regulation (e.g.,

Bandura, 1991; Carver & Sheier, 1990), the models converge around key features of a process that

sketches the paradigm. In essence, individuals regulate their attention and effort around goals that are

either self-set or influenced by the environment (e.g., what an instructor says, what a system prompts).

Feedback indicates the degree of discrepancy between current performance and the goal. Moderately

negative discrepancies are affectively unpleasant and generally prompt additional effort or a revision of

strategy to close the gap between performance and the goal. Substantially negative discrepancies are

very unpleasant and may prompt withdrawal of attention and effort -- the individual gives up. Positive

discrepancies are pleasant and may prompt coasting or the reallocation of attention to another goal. In

sum, self-regulation describes a cyclical, iterative process involving cognitive, behavioral, and affective

Psychological Fidelity 9 elements underlying skill acquisition. As a general model of learning and task performance, self-

regulation theory has amassed considerable support (e.g., Karoly, 1993).

Self-regulatory theories form the core of our research effort. They are the source of

fundamental psychological constructs, and structure the mechanisms used to understand the process of

learning, training outcomes, and adaptive performance. These constructs and mechanisms are the raw

material to be leveraged by instructional design.

Training and Instructional design. As we noted previously, the development of adaptive

performance skills is predicated on active learning. Active learning necessitates instructional

experiences that promote mindful processing, deliberate learning strategies, and deep comprehension

(Smith et al., 1997). Although a wide range of instructional tools intended to prompt active learning

have been proposed -- learner control (Steinberg, 1993), error-based training (Frese & Altman, 1989),

and mastery vs. performance states (Ames & Archer, 1988), these are simply isolated tools. An

integrative framework is needed to guide the use of these techniques to promote adaptive performance

skills (Kozlowski, 1998).

The framework we use to guide our research provides this integration (see Kozlowski, Toney,

Mullins, Weissbein, Brown, & Bell, 2001). First, it is designed to selectively influence the self-

regulatory process to influence learning, skill acquisition, and adaptive performance. In this regard, it

identifies a range of instructional design constructs that can leverage the regulatory process, including

the design of practice scenarios, the nature of goals and goal states, characteristics of feedback, and the

role of individual differences in combination with instructional features. Thus, it meshes well with the

core theoretical foundation of our approach that focuses on self-regulation. Second, it incorporates a

range of outcome constructs that are designed to tap the cognitive, behavioral, and affective aspects of

the regulatory process during skill acquisition. It also incorporates a distinction between routine training

performance and adaptive performance skills that must manifest under more difficult, dynamic, and

Psychological Fidelity 10 complex conditions. Thus, it provides guidance for research design, measurement, and evaluation in

our work. And, third, the research and application logic of the framework makes a distinction between

basic research designed to examine the pure effects of instructional constructs, and application-oriented

research designed to examine the effects of several instructional constructs that have been combined into

a training strategy. Basic research is focused on determining whether an instructional intervention has

effects, and whether those effects conform to theory. Research focused on the efficacy of a training

strategy is concerned with the combined effect of several interventions intended to have synergistic

effects. Thus, the framework provides a means to help bridge basic research findings to application.

Team development. Finally, because our work is intended to inform the development of

adaptive teams, we have to be sensitive to individual learning and development in the team context. It is

axiomatic that learning is an individual level phenomenon; teams can exhibit learning only as a

consequence of an emergence process that has its roots in individual learning (Kozlowski & Bell, in

press; Kozlowski & Klein, 2000). Our core model of self-regulation describes individual learning and

performance. However, the relevant question is how this Aself@ regulatory process unfolds when the

individual is part of a team that is striving to accomplish both individual and team goals? Teams provide

a context for individuals. Team level phenomena such as learning and performance are created by

individual interactions, but team level phenomena also have a significant influence on individuals. Thus,

the issues here are two-fold. First, an understanding of how teams normatively learn and develop

provides a basis for identifying what kind and when leverage can be best exerted in training. Second, an

appreciation of levels of analysis issues in research design and analysis allows us to tease apart

individual, individual in context, and team level effects (Kozlowski & Klein, 2000).

We draw on two related heuristics that focus on the development process for adaptive teams,

one of which addresses normative team learning and development (Kozlowski et al., 1999), and the

other of which addresses the role of leaders in this process (Kozlowski, Gully, McHugh, Salas, &

Psychological Fidelity 11 Cannon-Bowers, 1996; Kozlowski, Gully, Salas, & Cannon-Bowers, 1996). In the context of team

training, both frameworks suggest the use of training strategies that (a) shift from basic to strategic to

adaptive knowledge and skills, and (b) shift from individually focused self-regulation to team focused

regulatory processes over time. Thus, this theoretical leg helps to ensure that learning at the individual

level is linked to team learning, development, and performance.

A Psychological Fidelity Approach to Simulation-Based Training Design

As we noted in the introduction to this chapter, traditional approaches to simulation-based

training design focus on high physical fidelity to resolve the challenges of skill acquisition and transfer.

The logic of this approach is that realistic reproduction of the performance environment will implicitly

capture the essential psychological processes that underlie learning, performance, and generalization. In

contrast, the logic of the psychological fidelity approach is based on the use of theory to design

simulation experiences that evoke basic psychological processes relevant to the critical performance

requirements in the target task domain. Basic psychological processes can often be effectively evoked

with low physical fidelity simulationsBthat is, synthetic tasks--that are carefully designed to elicit

desired cognitive, behavioral, and affective responses. The psychological fidelity approach attempts to

resolve the issues of skill acquisition and transfer by explicitly using basic theory of fundamental

psychological processes of learning, motivation, and performance, and linking key constructs and

mechanisms underlying those processes to critical real-world task characteristics through simulation

design, scenario construction, and targeted assessment. Psychological fidelity then becomes a tool to be

employed with an appropriate research design strategy as discussed below. The basic elements of the

psychological fidelity approach are shown in Table 1.

Insert Table 1 about here

Psychological Fidelity 12 Principles of Psychological Fidelity

Effective utilization of the psychological fidelity approach begins with basic psychological

theory that is the source of relevant psychological constructs, cognitive mechanisms, and motivational

and behavioral processes. The second basic element of the approach is to generate information

regarding the cognitive requirements and behavioral performance specifications relevant to the task

domain of interest. Cognitive task analysis (CTA) and related techniques provide the means to generate

this essential information. The CTA, in combination with the critical constructs and processes of basic

theory, identify the parameters that are used to guide the selection or construction of a simulation or

scaled world. Note that the emphasis here is on developing or selecting a scaled world that allows basic

psychological constructs and processes relevant to the real-world task to be emulated in the simulation.

This element links together theory, information from the CTA, and features in the scaled world. The

last element focuses on the design of scenarios -- the practice experiences to which trainees are to be

exposed -- and the development of measures to capture critical psychological constructs, processes, and

outcomes. This element operationalizes the linkages among theory, the CTA, and the simulation.

Unlike the physical fidelity approach, scenarios are not necessarily designed to mimic real-world

events. Rather, the intent is to design scenarios that elicit theoretically based constructs and processes

identified by the CTA, and to develop measurement systems to track those constructs and processes as

they unfold during the simulation experience.

Research Design Strategies

Psychological fidelity provides us with a research tool. It ensures that basic psychological

constructs, processes, and outcomes -- linked to a real-world task -- are explicitly embedded in our

simulation and scenario design. Effective utilization is accomplished through research design strategies.

The basic elements of research design are shown in Table 2.



The basic elements of the research design strategies again begin with theory. In this instance,

theories of instructional design are used to link basic psychological theory to the construction of

experimental manipulations or interventions. There are two general research design strategies that are

used, depending on the purpose of the research, basic investigation or application. The first strategy is

appropriate for research designed to test basic theoretical propositions or to assess the potential of new

interventions. We characterize this strategy as research examining Apure@ manipulations that isolate the

effects of unitary constructs on underlying psychological processes or mechanisms involved in learning,

motivation, and performance.

The second strategy is appropriate for the design of instructional interventions and the

evaluation of their effectiveness in enhancing training outcomes and transfer. Instructional interventions

are rarely, if ever, based on pure constructs. Rather, training interventions are complex combinations of

several pure manipulations that are designed to work together synergistically to leverage underlying

psychological processes (Kozlowski, Toney et al., 2001). For example, the active learning intervention

mastery goals represents the combination of (a) a mastery frame or prime that emphasizes exploration

and understanding of the task domain, (b) learning objectives (rather than performance objectives), and

a sequence of practice that shifts attention from basic knowledge and skills to more complex knowledge

and strategic skills as learning progresses (e.g., Kozlowski, Gully, Brown, Salas, Smith, & Nason,

2001). This second strategy is predicated on basic research that demonstrates the promise of

manipulations based on pure constructs. A theory of instructional design is used to guide the

combination of these single constructs into a complex manipulation or training strategy. Research then

evaluates the impact on key processes involved in learning, motivation, and performance to ascertain

the effectiveness of the training strategy. This is an essential step for translating the results of basic

research employing psychological fidelity and moving it toward application in higher physical fidelity

Psychological Fidelity 14 simulations, and in real-world training design. The goal of this entire process is to develop a

theoretically based, research proven, and applications relevant set of training principles and tools.

An Illustration of the Psychological Fidelity Approach

The best way to explain the psychological fidelity approach is to ground it in a specific research

application. As shown in Table 3, we illustrate how the basic principles of psychological fidelity are

applied to our research.


Psychological Fidelity

Basic Theory. The foundation of our research is based on models of self-regulation and action

initiation. These models enjoy broad-based support in the educational and psychological literatures as

explanations of learning, motivation, and performance. Thus, they are a rich source of constructs,

cognitive mechanisms, and motivational and behavioral processes. A selective tour of characteristics we

examine in our research follows.

Although training design is often consistent for all trainees, it is well known that trainee

individual differences can affect the nature and quality of self-regulation. For example, general

cognitive ability, g, represents the cognitive resources an individual can devote to learning; g is the

single best predictor of training effectiveness (Ree & Earls, 1991). We examine the direct effects of

cognitive ability on learning, and also examine potential interactions with other non-cognitive individual

differences and intervention constructs.

More specifically relevant to self-regulation are individual differences in goal orientation.

Theory suggests that goal orientation, learning or performance, influences the way individuals approach

learning situations (Button, Mathieu, & Zajac, 1996; Dweck, 1986; Nicholls, 1984). Learning

orientation represents an adaptive style in which the individual strives for achievement, treats mistakes

and negative feedback as learning opportunities, and persists in the face of obstacles. In contrast,

Psychological Fidelity 15 performance orientation represents a maladaptive style in which the individual seeks to demonstrate

competence, avoids making mistakes, and withdraws from the task in the face of difficulty or negative

feedback. Clearly, learning orientation is potentially relevant to the development of adaptive

performance skills.

Self-regulatory processes are fundamentally concerned with managing goal-performance

discrepancies. Therefore, relevant constructs include the types of goals (e.g., to master skills, or to

perform well) and goal levels (e.g., easily attainable goals, or difficult goals) individuals choose to

pursue. Factors that may influence goals include individual differences or interventions designed to

induce a particular goal state (e.g., learning or performance). Additional constructs implicated by the

self-regulatory process include goal commitment, performance monitoring, and feedback interpretation,

which capture the quality of self-regulation. Affective, cognitive, and behavioral outcomes include self-

efficacy and self-satisfaction (affective), basic and strategic knowledge (cognitive), and basic and

adaptive performance (behavioral).

The cognitive, motivational, and behavioral mechanisms of self-regulation are captured by

changes in the states of constructs noted above and by additional characteristics. For example, cognitive

mechanisms can be ascertained by examining the application of meta-cognitive strategies and the

allocation of attention, thereby allowing us to address such issues as: Do trainee strategies adjust

appropriately in response to negative feedback? Do trainees allocate cognitive resources to high task

priorities? Learning in the self-regulation paradigm is keyed to reducing goal-performance

discrepancies, managing affect, especially early in skill acquisition when the gap between current

performance and the end goal is substantial, and preventing premature cognitive withdrawal.

Motivational and behavioral processes can be captured by examining effort devoted to study and

practice, and whether study and practice is focused on key skills in an appropriate instructional

sequence (Bell & Kozlowski, 2002). At the team level, cooperation and coordination of effort, and the

Psychological Fidelity 16 capability of the team to acquire basic, strategic, and adaptive performance skills are critical to

evaluation of the theoretical model.

Cognitive task analysis. The CTA is the source of cognitive requirements and performance

specifications relevant to the task domain of interest. In our approach, it links the constructs and

processes of basic psychological theory to the real-world task, and provides the necessary information

to select or construct a simulation (Kozlowski & DeShon, 1999). Thus, it is an essential link in ensuring

the psychological fidelity of the simulated world. Recognize, however, that the focus is not on

representing realism. Rather it is on abstracting key characteristics of the performance domain and its

cognitive requirements that are relevant to a basic theory of psychological functioning (Cannon-Bowers

& Salas, 1997). Thus, our approach makes for a somewhat different emphasis on the information that is

abstracted from the CTA.

Several CTAs of military DDM task domains (e.g., Johnston, Smith-Jentch, & Cannon-

Bowers, 1997; MacMillan, Serfaty, Young, Klinger, Thordson, Cohen, & Freeman, 1998) have

identified defining features that are relevant to the dynamic regulation of attention and effort that is at

the core of self-regulation theory. For example, DDM tasks capture relations with a task environment

that is in a state of flux such that information about the nature of the situation emerges over time. Thus,

the task often lacks clear structure, is ambiguous, and may present conflicting information as to the true

nature of the situation. As the apparent situation emerges, the decision maker may be faced with

conflicting or incompatible goals. Evolving assessments of the situation as it unfolds necessitates rapid

diagnosis of the situation, prioritization of goals and potential actions, and implementation of

appropriate task strategies.

The cognitive and performance requirements relevant to such tasks include cognitive resources

to manage the substantial information processing and decision-making demands. Another critical

requirement is the capability to develop, maintain, and adapt an awareness of the overall situation and

Psychological Fidelity 17 its current priorities. Based on that assessment, the decision maker must be able to select an appropriate

task strategy, monitor its effectiveness, and revise the strategy as necessary. In a team context, the

decision maker must monitor teammate actions, be ready to backup overloaded teammates, and

coordinate concerted team action.

Simulation. In the psychological fidelity approach, the central factor in the selection or

construction of a synthetic task is that it must provide a means to capture key CTA requirements and

critical constructs and processes from basic psychological theory. Ensuring high psychological fidelity

with respect to DDM tasks was a central goal in the development of TEAMSim (Team Event-based

Adaptive Multilevel Simulation).

TEAMSim is a PC-based, radar tracking simulation that can be configured to emulate virtually

any radar-tracking task at the individual or team level. The simulation uses scripted events that unfold

in real time to provide a shifting and emergent situation that demands adaptability. In our research, the

simulation is configured to emulate three-person teams of AWACS Weapons Directors. Three person

teams are seated at simulated radar consoles that present multiple, dynamically interacting contacts.

Targets possess different characteristics and threat profiles, and exhibit different patterns of movement.

Participants must make three identification decisions and then render an overall decision for the targets.

In addition, complex task relations are embedded in the scenario design that necessitate shifts in task

priorities and strategies, and the allocation of individual and team resources. TEAMSim provides

participants with a dynamic, self-contained, and completely novel task environment that is appropriate

for examination of complex skill acquisition and adaptability.

There were two primary design challenges in the development of TEAMSim. The first was to

construct a simulation using dynamic event-based scenarios to provide a complex, shifting, and

emergent task. Simulations that present a series of single decision situations lack the dynamics,

complexity, and emergent properties of DDM, and are not well suited to examining skill acquisition and

Psychological Fidelity 18 adaptability. The second challenge was to devise a theoretically based measurement system to allow

both individual and team constructs to be examined.

There are several basic information processing and decision-making features constructed into

TEAMSim. Each operator is seated in front of a simulated radar display that is dominated by a radar

plot. The plot is separated into three sectors of responsibility, one for each of the three-team members.

Current location is indicated by the center of the plot, which is surrounded by a designated defense zone

at 10 nautical miles (NM). Compass bearings around the plot indicate target location and bearing.

Scripted radar contacts move about the plot. The upper left corner displays elapsed time for the scenario

(counting down to zero). The lower left corner displays radar range. Range can be changed by using a

pull-down menu. The lower right corner displays the hooked track number. Contacts are selected with a

mouse click; each target has a unique track identification number. The upper right corner displays a

cumulative score. It can be selected to display individual, team, both, or no scores.

Once the operator has "hooked" or highlighted a contact, he/she collects cue information using

pull-down menus. Contact information is completely configurable. Up to five information cues can be

used for each of three subdecisions. Cues can be made accurate, ambiguous, or conflicting. Once the

operator has rendered each of the three subdecisions, a final decision must be made. The engage

decision can range from clearing non-threatening contacts to shooting hostile ones (e.g., clear, ignore,

monitor, warn, lock-on, shoot).

Aside from the information processing demands, the key features of TEAMSim that make it a

useful research and training design platform is the capability to script dynamic, ambiguous, and

emergent event-based scenarios. Such scenarios present the research participant or trainee with a task

environment that entails shifting goals and task priorities, the need to monitor and coordinate with one=s

teammates, and the need to adapt to meet emerging task demands. These characteristics are realized

through scenario design, which is discussed below.


Scenario design and measurement. Simulation features and their configuration provide the raw

material for scenario design. As shown in Table 3, it is in the design of scenarios and development of

measurements that basic psychological constructs and processes and CTA requirements are

operationally linked (Dwyer, Oser, Fowlkes, & Lane, 1997). The problem situation that emerges for

the individual or team is scripted by the experimenter. Primary scenario design features include target

configuration and distribution, course and speed, pop-ups, and defensive perimeter intrusions. Target

configuration and distribution controls what the situation looks like as the problem begins. It can look

benign, with more a threatening situation developing out of the selected radar range. Or it can look very

active, but the overall situation might be even more threatening at longer range. Target course and

speed determines how the situation evolves and emerges. The movement of each target is scripted. By

coordinating the timing and movement of multiple targets, the experimenter can create "events" that call

for particular strategies or actions. Pop-ups are targets that suddenly appear on the display. They

simulate an aircraft emerging under radar cover or a defective sensor. Because they appear suddenly,

they have the potential to immediately change the strategic situation and the appropriate response. One

or more defensive perimeters can be specified; as a matter of policy contacts are not to penetrate these

perimeters. Thus, contacts that threaten to penetrate defensive perimeters necessitate engagement and

drive task strategies. By coordinating intrusive targets, pop-ups, and the number and value of defensive

perimeters, the experimenter can script challenging situations that demand the need to establish

priorities, employ strategies, and make adjustments as the scenario unfolds. Data can be tracked down

to the target level (latencies, decisions, engagements, decision modifications), which provides the

capability to track responses to specific scenario events. Data can be examined at the team level,

individual by position, and targets within individuals.

TEAMSim can be configured at the individual or team levels of analysis. At the individual

level, the simulation is focused on the acquisition of cognitive and behavioral skills, and their relevance

Psychological Fidelity 20 to adaptability when the underlying rules or principles of the problem situation change. Here the

primary design features include task workload and complexity (Wood, 1986). Feedback features,

scoring algorithms, and the decision structure can also be varied. A variety of antecedents, process

indicators, and outcomes are assessed by the simulation and with supplementary measurements. At the

team level, the simulation is focused on processes relevant to team coordination and adaptability. Here

the primary design features include coordination and communication demands, which are varied by

distributing access to information and defining the team member roles in the decision structure. It

should be noted that communication processes are natural and largely occur through the use of verbal

statements. There is no computer mediation, as computer mediation is known to affect communication

processes (Coovert & Thompson, 2001). Finally, the individual and team level components of the

overall decision model can be mathematically modeled as distinct elements. In this sense, TEAMSim

provides a true multilevel capability for modeling team performance.

Research Paradigm

The basic elements of the psychological fidelity approach are integrated and realized through

research design. As shown in Table 4, the research design process is guided by (a) theories of

instructional design, with the intent of developing manipulations that have effects on key constructs and

mechanisms specified by basic theory, and (b) theories of training effectiveness and evaluation, with the

goal of ascertaining the effects of experimental training manipulations on a broad array of process

indicators and outcomes. In other words, research design applies the basic elements of psychological

fidelity -- basic theory, CTA, simulation features, and scenario design and measurement -- to design

interventions and to assess their impact on basic learning processes and performance outcomes. Thus,

this process is designed to yield theoretically driven, applications relevant, and generalizable principles

of training design for the domain in question.



Instructional design. As we noted previously, there are two general strategies that can be

employed in this process: one focused on basic research and the other focused on the design of training

strategies intended for application. The first approach involves basic research that examines the effects

of pure or isolated constructs that have potential promise as training interventions. The effects of such

constructs on influencing learning processes and performance in ways consistent with underlying theory

provides a means to evaluate both their potential as training interventions and the utility of the

underlying theoretical model driving the research. The second approach is intended to develop training

strategies with the potential for application. It pulls together pure constructs that have shown promise in

basic research, and combines them in an effort to create a synergistic and effective instructional

intervention -- one that influences key processes involved in learning, motivation, and performance.

These two strategies are designed to be complementary. Basic research develops and tests promising

instructional constructs and principles; training design research draws on that foundation to develop

tools and techniques that move closer to application.

Training effectiveness. Research design for both strategies is driven by theory pertaining to

training effectiveness and evaluation. These theories structure several aspects of our paradigm. First,

Kraiger, Ford, and Salas (1993) make a strong case for a multidimensional conceptualization of training

outcomes. Their review of the cognitive, educational, and organizational literatures demonstrated the

pervasive tendency to operationalize training effectiveness is simple terms. Oftentimes, simple measures

of declarative knowledge are used as learning indicators. In many other instances, simple global

measures of task performance are used as indicators of Alearning.@ Thus, our research conceptualizes

the effects of training multidimensionally. Second, Kozlowski (1998) emphasizes the importance of

temporal processes during individual learning and team development. Although it is common to assess

learning curves in studies of the acquisition of simple motor skills, developmental assessment of

learning is not common for more complex and cognitively loaded task domains such as those used in

Psychological Fidelity 22 simulation-based training. Thus, our research design tracks change in multiple measures over time.

Third, Kozlowski and Salas (1997) indicate the need for multilevel assessment -- individual and team --

in team training contexts (see also Kozlowski & Klein, 2000). Research has shown a marked tendency

to focus either on the team level or on the individual level. However, in team training, teams provide

contexts that influence the nature and progress of individual learning. Thus, our research design

accounts for both the influence of the team on individual learning, and the influence of individual level

skill development on team performance. And, fourth, Kozlowski, Toney et al. (2001) emphasize the

importance of skill generalization or performance adaptability in the assessment of training

effectiveness. Outcomes assessed at the end of training are typically predicated on reproduction of key

knowledge and skill emphasized in training. However, from a transfer perspective, we are interested

not merely in skill replication and maintenance, but more importantly in skill generalization. Skill

replication is an important training goal. However, if skill replication is the primary target of training

design, then simulation-based training for complex dynamic task domains will have to provide scenarios

and practice for every conceivable (and inconceivable!) contingency. Such a strategy is obviously

impossible. Hence, our approach is focused on developing self- and team-regulatory processes and

relying on these basic psychological mechanisms to provide the means for trainees to generalize

knowledge and trained skills to situations not encountered during training.

Research model. These basic elements of research design are incorporated in our model, a

schematic representation of which is shown in Figure 1. The model is consistent with the Kozlowski,

Toney et al. (2001) model of instructional design for simulation-based training. It is organized around

four sets of primary factors. Antecedents include Research Factors (i.e., pure constructs or complex

interventions) and Individual Differences in cognitive abilities (e.g., g, metacognitive skills) and

dispositional traits (e.g., learning and performance orientation). Basic Psychological Processes include

constructs and mechanisms of individual self-regulation. They encompasses such factors as goal level,

Psychological Fidelity 23 type, and commitment; goal-performance trajectories; and metacognitive strategies, cognitive resource

allocation, and cognitive attention/withdrawal. Because our model is explicitly multilevel (Kozlowski &

Klein, 2000), there are analogues of key regulatory factors that operate at the team level. These factors

include goal characteristics, goal-performance trajectories, and team task strategies. Depending on the

research strategy of the experiment in question (basic or development), we may also assess effort and

team process indicators (see Table 3). Training Effectiveness is assessed with respect to individual

cognitive outcomes of basic and strategic knowledge, behavioral indicators of basic and strategic

performance, and affective measures of self-efficacy and performance satisfaction. Again, the model

incorporates team level analogues. Finally, Transfer and Generalization are defined operationally by the

capability of individuals and teams to adapt knowledge and skills acquired during training to a more

difficult, more dynamic, and more complex version of the simulation. This is captured by the retention

of basic skills and the adaptation of strategic skills.

Insert Figure 1 about here

Illustration. The application of this research model can be illustrated through examples drawn

from our research stream. Our application of the basic research strategy draws on two primary theories

of training and development, both of which include self-regulation as a basic psychological process. The

theory of instructional design (Kozlowski, Toney et al., 2001) provides the general architecture and

logic of our research model, forms the basis for the two training strategies, and identifies potential

research factors and individual differences of interest. The theory of team compilation and development

(Kozlowski et al., 1999) provides a means to extend the model to the team level and to identify

potential research factors relevant when individuals are trained in a team context.

Examples of the basic research strategy within this framework address three sets of pure

constructs as a means to understand their impacts on individual and team regulation (see Table 4). We

previously noted that the educational literature has suggested that individual differences in goal

Psychological Fidelity 24 orientation reflect learning styles that can yield quite different effects with respect to learning,

motivation, and performance. Learning oriented individuals seek to develop mastery, making effective

use of feedback and persisting in the face of difficulties. In contrast, performance oriented individuals

seek to demonstrate competence and to avoid new situations where they lack skills. They try to

minimize mistakes and to withdraw from the task in the face of negative feedback--inevitable aspects of

learning complex tasks such as those underlying simulation based training. Although the literature has

conceptualized these characteristics as stable traits (and we do assess them as traits), they can also be

conceptualized as situationally induced states. Indeed, training situations often inadvertently create a

performance orientation in trainees by trying to minimize errors and emphasizing high performance.

Thus, learning vs. performance-oriented states can be induced and theoretical effects on the quality of

self-regulation can be examined. Without going into detail, theory would predict that trainees with

situationally induced learning states, relative to performance states, should have more effective

regulatory processes with positive effects on learning, training performance, and adaptability

(Kozlowski, Gully et al., 2001; Kozlowski, Toney et al., 2001). Additional questions concern the

manifestation of goal orientation at the team level (e.g., DeShon, Milner, Kozlowski, Toney, Schmidt,

Wiechmann, & Davis, 1999).

One of the challenges of team training design is determining the appropriate focus of regulatory

activity, individual or team? Normative theory pertaining to the compilation and development of team

performance skills asserts the necessity for individuals to first acquire basic proficiency on their

performance skills before they are capable of shifting their attention to team level skill integration

(Kozlowski et al., 1999). Other theories of team building emphasize the necessity of developing the

team as a holistic entity. Thus, the former theory suggests a developmental sequence that first

emphasizes individual level regulation that then shifts to team level regulation, whereas the latter

perspective suggests that team level regulation should be emphasized throughout training. Preliminary

Psychological Fidelity 25 findings indicate support for shifting the focus of regulatory activity from individual goals to team goals

across training (DeShon, Kozlowski, Schmidt, Wiechmann, & Milner, 2001).

Feedback is another research factor with high potential to influence the quality of regulatory

processes. Although conventional wisdom suggests that feedback is essential to good training design,

recent meta-analytic findings indicate that the effects of feedback on learning and performance are

remarkably inconsistent (Kluger & DeNisi, 1996). Thus, the real issue is how to effectively configure

feedback and incorporate it into training design. One of the ambiguities regarding feedback and team

training is at what level should feedback be targeted -- the individual, the team, both levels?

Remarkably, this issue has received virtually no research attention. Self-regulation theory places

feedback as a central factor in the learning process. It is the source of information regarding goal-

performance discrepancies, and hence prompts individual strategy formulation intended to reduce

discrepancies in learning situations. Applying a multilevel perspective, we can speculate that the level

of feedback delivery will influence the focus of regulatory activity. Direct it at the individual level, and

individuals should regulate around their goals at the expense of attention to team goals. Direct it at

teams, and it should facilitate team level processing with less attention devoted to regulation around

individual goals. Theory is ambiguous regarding the effects of directing feedback at both levels; it may

prompt superior individual and team regulation, or it may simply overwhelm the trainee with

conflicting demands undermining performance at both levels. Early research results suggest the latter

effect. Team members receiving both individual and team feedback were not able to simultaneously

maximize both individual and team performance. Teams receiving only team feedback performed

substantially better than teams receiving either individual and team feedback or just individual feedback.

Similarly, individuals receiving only individual feedback had substantially higher levels of individual

performance relative to either of the other two conditions (DeShon, Kozlowski, Wiechmann, Milner,

Davis, & Schmidt, 2000).


The common feature across all three of these examples is that they address the effects of pure

constructs on basic psychological processes involved in individual level and/or team level regulation.

They are theory driven constructs, examined singly or in considered combination, with the intent of

modeling effects on regulatory processes, learning, and performance. Findings from basic research such

as this provides input for the second research strategy.

An example of the application of the second research strategy, designed to combine pure

constructs to create an effective training strategy, is illustrated by research on goal sequence, goal type,

and goal consistent feedback. Instructional design theory and research indicate that appropriate

sequencing of learning goals from more basic to more complex is an important aspect of effective

instructional design (Gagne, 1970). We earlier discussed expected differences of the effects of

performance-oriented goals relative to learning or mastery oriented goals. Mastery oriented goals, that

is, goals that emphasize mastering key learning objectives, should yield superior learning relative to

performance-oriented goals. Finally, self-regulation theory would predict that feedback needs to be

consistent with goals to support the self-regulatory process. Remarkably, there has been little research

examining goal-feedback consistency. Moreover, some research has shown that mastery goals are

superior to performance goals, even when used with performance feedback (Kozlowski, Gully et al.,

2001). Regardless, these three pure constructs -- goal sequence, goal type, and feedback consistency --

can be combined to create training different strategies: sequenced mastery goals/feedback vs. sequenced

performance goals/feedback. Without going into detail, sequenced mastery goals/feedback would be

expected to yield superior regulatory processes, learning, and performance. Early evidence at the

individual level supports that prediction (Mullins, Kozlowski, Toney, Brown, Weissbein, & Bell,

1999).

The essential element of this example is the way in which different training manipulations are

combined together to create a more complex strategy. Here, the constructs to be combined have to have

Psychological Fidelity 27 some common underpinnings to guide their combination. In the psychological fidelity approach, that

common underpinning is basic theory; it guides what should be combined and how it should be

combined. The result is data to support an effective training strategy; a strategy that is theoretically

based and applications relevant; a strategy that by design has the potential to generalize to alternative

simulation platforms -- at the same or higher fidelity -- to the extent that they entail the same cognitive

and performance requirements and the same underlying psychological processes.

Discussion and Conclusion

We noted at the onset of this chapter that issues of transfer effectiveness pose a critical

challenge for training researchers and designers. Moreover, training design for complex DDM tasks

must not only be sensitive to transfer as skill reproduction, but more importantly it must be concerned

with training design that enhances transfer as knowledge and skill generalization; that is, as

adaptability. This focus on enhancing the adaptive capabilities of individuals and teams through training

design is at the center of our research. The principles of psychological fidelity form the core of our

strategy for accomplishing this goal.

The dominant approach to meeting the challenges posed by the problem of training transfer has

been to focus on high physical fidelity simulations. High fidelity simulations are designed to emulate the

complexity of real-world tasks, without entailing the costly consequences of failure that are often

associated with such systems. It is an elegant strategy because it seeks to minimize the transfer problem

by closing the gap between training and the real-world task. Training and task performance merge. It is

an effective strategy that has served the training community wellBand it will continue to do so.

Nevertheless, in spite of its many strengths, high physical fidelity simulation possesses several

limitations as a training design strategy. It requires significant investment in dedicated facilities and

hardware. It must be staffed by expert trainers and coaches. There must be support facilities for

trainees. There must be coverage of trainee, travel, maintenance, and lost-work costs. Moreover, the

Psychological Fidelity 28 considerable investment required to establish and maintain such facilities almost always ensures that

training demands will exceed the capacity of the training system. Such facilities, then, often become

major bottlenecks for training. Thus, cost-effectiveness and efficiency become troublesome factors for

training systems constructed around high fidelity simulations in a time of increasingly tight training

budgets and correspondingly greater demands for high-level skills.

These challenges and constraints are driving the push for new, more cost-effective, and more

efficient training technologies. Increased computer power, declining costs, and enhanced connectivity

are making possible a host of new and advanced training technologies that allow training to be

distributed. The potential of these technologies is creating a move to push training out of centralized

facilities to far-flung distributed systems (Kozlowski & Bell, 2002). By distributing training across

computer networks, high facilities and maintenance costs are eliminated. Distributing training, by

necessity, also entails a push away from high-cost high physical fidelity simulation systems and a shift

toward lower cost low fidelity systems that run on common PC platforms.

Distributed training is a general label used to describe training systems in which trainees are

geographically separated from an instructor and/or other trainees, and can assume two primary forms.

One form of distributed training uses advanced video-conferencing and communication technologies to

enable an instructor to hold class for trainees in geographically remote locations. A second form of

distributed training--and the form relevant to our training research--focuses on interactive multi-media

applications in which information (e.g., web-based training) and simulation-based practice can be

provided over the Internet or internal intranets. This form of distributed training has the potential to

enable a new architecture for training design for complex military DDM tasks (e.g., command and

control). Multiple, geographically dispersed trainees (as individuals, teams, and teams of teams) will be

able to engage in sophisticated simulation-based practice to hone their basic skills and to develop

higher-level strategic and adaptive skills.


The cost logic of this shift to a distributed training design strategy is compelling, but will it be

effective? Will it meet the challenges posed by the transfer problem? In our view, the critical challenge

to realizing the promise of distributed training is the need for effective instructional principles and

strategies--grounded in psychological theory and research--to guide the design and application of system

features and capabilities. The technology is merely a medium for delivering information or experience;

it is not instruction per se. The psychological fidelity approach to training research and design is

predicated on meeting this challenge; on realizing the potential of this new generation of training tools;

on developing theoretically-based and applications-relevant instructional principles that enhance skill

acquisition and adaptability for individuals and teams.

Psychological Fidelity 30 References

Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students' learning

strategies and motivational processes. Journal of Educational Psychology, 80, 260-267.

Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63-105.

Bandura, A. (1991). Social cognitive theory of self-regulation. Organizational Behavior and Human Decision Processes, 50, 248-287.

Bandura, A., & Wood, R. (1989). Effect of perceived controllability and performance standards on self-regulation of complex decision-making. Journal of Personality and Social Psychology, 56, 805-814.

Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn? A

taxonomy for far transfer. Psychological Bulletin, 128, 612-637. Bell, B. S., & Kozlowski, S. W. J. (2002). Adaptive guidance: Enhancing self-regulation,

knowledge and performance in technology-based training. Personnel Psychology, 55, 267-306. Button, S. B., Mathieu, J. E., & Zajac, D. M. (1996). Goal orientation in organizational

research: A conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67, 26-48.

Cannon-Bowers, J. A., & Salas, E. (1997). A framework for developing team performance measures in training. In M. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement (pp. 45-62). Mahwah, NJ: LEA.

Carver, C. S. & Sheier, M. F. (1990). Principles of self-regulation: Action and emotion. In E. T.

Higgins and R. M. Sorrentino (Eds.), Handbook of motivation and cognition (Vol. 2, pp.3-52). New York: Guilford.

Coovert, M. D., & Thompson, L. F. (2001). Computer supported cooperative work: Issues and

implications for workers, organizations, and human resource management. Thousand Oaks, CA: Sage. DeShon, R. P., Kozlowski, S. W. J., Schmidt, A. M., Wiechmann, D., & Milner, K. R. (2001,

April). Developing team adaptability: Shifting regulatory focus across levels. In S. W. J. Kozlowski & R. P. DeShon (Chairs), Enhancing team performance: Emerging theory, instructional strategies, and evidence. Symposium presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA.

DeShon, R. P., Kozlowski, S. W. J., Wiechmann, D., Milner, K. R., Davis, C. A., &

Schmidt, A. M. (April, 2000). Training and developing adaptive performance in teams and individuals. In S. W. J. Kozlowski (Chair), Developing complex adaptive skills: Individual- and team-level training strategies. Symposium to be presented at the 15th Annual Conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.


DeShon, R. P., Milner, K. R., Kozlowski, S. W. J., Toney, R. J., Schmidt, A., Wiechmann, D., & Davis, C. (1999, April). The effects of team goal orientation on individual and team performance. In D. Steele-Johnson (Chair), New Directions in goal orientation research: Extending the construct, the nomological net, and analytic methods. Symposium conducted at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.

Druckman, D., & Bjork, R.A. (1991). In the mind=s eye: Enhancing human performance. Washington, DC: National Academy Press.

Dweck, C.S. (1986). Motivational processes affecting learning. American Psychologist, 41, 1040-1048.

Dwyer, D. J., Oser, R. L., Fowlkes, J. E., & Lane, N. E. (1997). Team performance measurement in distributed environments: The TARGETS methodology. In M. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement (pp. 137-154). Mahwah, NJ: LEA.

Flavell, J.H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34, 906-911.

Frese, M., & Altmann, A. (1989). The treatment of errors in learning and training. In L. Bainbridge & S. A. R. Quintanilla (Eds.), Developing skills with information technology (pp. 658-682). New York: Wiley.

Gagne, R. M. (1970). Conditions of learning. New York: Holt, Rinehart, & Winston.

Goldstein, I. L. (1993). Training in organizations: Needs assessment, development, and evaluation. Pacific Grove, CA: Brooks/Cole.

Holyoak, K.J. (1991). Symbolic connectionism: Toward third-generation theories of expertise. In K.A. Ericsson & J. Smith, Toward a general theory of expertise (pp. 301-336). Cambridge, UK: Cambridge University Press.

Johnston, J. H., Smith-Jentch, K. A., & Cannon-Bowers, J. A. (1997). Performance measurement tools for enhancing team decision-making training. In M. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement (pp. 311-327). Mahwah, NJ: LEA.

Karoly, P. (1993). Mechanisms of self-regulation: A systems view. Annual Review of Psychology, 44, 23-52.

Kluger, A.N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.

Kozlowski, S. W. J. (1998). Training and developing adaptive teams: Theory, principles, and research. In J. A. Cannon-Bowers & E. Salas (Eds.), Decision making under stress: Implications for training and simulation (pp. 115-153). Washington, DC: APA Books.


Kozlowski, S. W. J., & DeShon, R. P. (1999, June). TEAMSim: Examining the development of basic, strategic, and adaptive performance. Presented at the International Synthetic Task Development Conference/Scaled Worlds: Current Issues in Simulation-Based Human Performance Research, University of Georgia, Athens, GA.

Kozlowski, S. W. J., Gully, S. M., McHugh, P. P., Salas, E., & Cannon-Bowers, J. A. (1996). A dynamic theory of leadership and team effectiveness: Developmental and task contingent leader roles. In G. R. Ferris (Ed.), Research in personnel and human resource management (Vol. 14, pp. 253-305). Greenwich, CT: JAI Press.

Kozlowski, S. W. J., Gully, S. M., Nason, E. R., & Smith, E. M. (1999). Developing adaptive teams: A theory of compilation and performance across levels and time. In D. R. Ilgen & E. D. Pulakos (Eds.), The changing nature of work performance: Implications for staffing, personnel actions, and development (pp. 240-292). San Francisco: Jossey-Bass.

Kozlowski, S. W. J., Gully, S. M., Salas, E., & Cannon-Bowers, J. A. (1996). Team leadership and development: Theory, principles, and guidelines for training leaders and teams. In M. Beyerlein, D. Johnson, & S. Beyerlein (Eds.), Advances in interdisciplinary studies of work teams: Team leadership (Vol. 3, pp. 251-289). Greenwich, CT: JAI Press.

Kozlowski, S. W. J., Gully, S. M., Brown, K. G., Salas, E., Smith, E. A., & Nason, E. R. (2001). Effects of training goals and goal orientation traits on multi-dimensional training outcomes and performance adaptability. Organizational Behavior and Human Decision Processes, 85, 1-31.

Kozlowski, S. W. J., & Klein, K. J. (2000). A multilevel approach to theory and research in

organizations: Contextual, temporal, and emergent processes. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations: Foundations, extensions, and new directions (pp. 3-90). San Francisco, CA: Jossey-Bass.

Kozlowski, S. W. J., & Salas, E. (1997). An organizational systems approach for the implementation and transfer of training. In J. K. Ford, S. W. J. Kozlowski, K. Kraiger, E. Salas, & M. Teachout (Eds.), Improving training effectiveness in work organizations (pp. 247-287). Mahwah, NJ: Lawrence Erlbaum Associates.

Kozlowski, S. W .J., Toney, R. J., Mullins, M. E., Weissbein, D. A., Brown, K. G., & Bell, B. S. (2001). Developing adaptability: A theory for the design of integrated-embedded training systems. In E. Salas (Ed.), Advances in human performance and cognitive engineering research (pp. 59-123). Amsterdam: JAI/Elsevier Science.

Kraiger, K., Ford, J.K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328.

MacMillan, J., Serfaty, D., Young, P., Klinger, D., Thordson, M., Cohen, M., & Freeman, J. (1998). A system to enhance team decision-making performance. Woburn, MA: Aptima, Inc.


Mullins, M. E., Kozlowski, S. W. J., Toney, R. J., Brown, K. G., Weissbein, D. A., & Bell, B. S. (1999, April). Adaptive performance: Mastery versus performance goals and feedback consistency. Paper presented at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.

Nicholls, J.G. (1984). Achievement motivation: Conceptions of ability, subjective experience, task choice, and performance. Psychological Review, 91, 328-346.

Orasanu, J., & Connolly, T. (1993). The reinvention of decision making. In G. Klein, J. Orasanu, R. Calderwood, & C. Zsambok (Eds.), Decision making in action: Models and methods (pp. 3-20). Norwood, NJ: Ablex Publishing.

Ree, M. J., & Earles, J. A. (1991). Predicting training success: Not much more than g. Personnel Psychology, 44, 321-332.

Smith, E. M., Ford, J. K., & Kozlowski, S. W. J. (1997). Building adaptive expertise: Implications for training design. In M. A. Quinones & A. Dudda (Eds.), Training for a rapidly changing workplace: Applications of psychological research (pp.89-118). Washington, DC: APA Books.

Steinberg, E. R. (1989). Cognition and learner control: A literature review, 1977-1988. Journal of Computer-Based Instruction, 16, 117-121.

Wood, R. E. (1986). Task complexity: Definition of the construct. Organizational Behavior and Human Decision Processes, 37, 60-82.

Psychological Fidelity 34 Table 1 Basic Elements of the Psychological Fidelity Approach Basic Elements

Function

Basic Psychological Theory

Source of: # psychological constructs # cognitive mechanisms # motivational and behavioral processes

Cognitive Task Analysis (CTA)

Grounds the task domain in its underlying # cognitive requirements # performance specifications

Simulation Selection or Construction

Links: # basic theory # CTA # scaled world

Scenario Design and Measurement

# Translate critical CTA requirements to specific scenario features # Operationalize measures to capture critical psychological constructs, processes, and outcomes

Psychological Fidelity 35 Table 2 Research Design Strategies Basic Elements

Function

Theory of Instructional Design

Guides the design of training manipulations to leverage key constructs/mechanisms specified by basic psychological theory. Two research strategies: # Research on Apure@ manipulations (isolated

constructs) designed to assess the impact on relevant constructs and processes. Appropriate for basic research to evaluate basic theory or to assess new manipulation tools.

# Research on complex training strategies (a

combination of several Apure@ constructs) designed to optimize the instructional impacts on underlying processes. Appropriate for research to assess training and/or transfer effectiveness.

Theory of Training Effectiveness and Evaluation

Guides research design features: # Assessment of multidimensional training

outcomes to capture cognitive, affective, and behavioral domains.

# Specifies developmental assessments at

multiple time points during skill acquisition. # Specifies mulitlevel assessment -- individual

and team -- in team training contexts. # Specifies assessment of skill generalization

and adaptability.

Psychological Fidelity 36 Table 3 Illustration of the Psychological Fidelity Approach

Basic Elements

Function

Basic Psychological Theory: # Self-Regulation and Action Initiation

Source of: # psychological constructs

individual differences in cognitive ability and learning and performance goal orientation, variation in goal types, goal levels, goal commitment, performance monitoring, feedback interpretation/attribution, self-efficacy, self-satisfaction, knowledge, performance, adaptive performance

# cognitive mechanisms

metacognitive strategies, cognitive resource allocation, trajectory of performance-goal discrepancies, affective reactions, cognitive attention/withdrawal

# motivational and behavioral processes

effort devoted to study and practice, focus of study and practice (content, sequence), cooperation and coordination, basic, strategic, and adpative performance

Cognitive Task Analysis (CTA): # DDM Tasks

- ill structured, emergent - incompatible/conflicting goals - shifting goal priorities - ambiguous/conflicting cues - time pressure - multiple actors - stress

Grounds the task domain in its underlying: # cognitive and performance requirements

- cognitive resources - information processing/decision-making - situational awareness - planning task strategies - monitoring effectiveness of task strategies - revising, adapting task goals/strategies - monitoring teammates - coordinating team action


Basic Elements

Function

Simulation Selection or Construction # Simulation Features: - scripted, event-based scenarios - dynamic, emergent situations - variable information processing demands - variable cue ambiguity and conflict - variable target priorities - shifting task strategies and trade-offs - variable difficulty, dynamics, and complexity - data to assess situational awareness, priorities, and strategies - data to assess mutual performance monitoring and coordination - data to assess basic, strategic, and adaptive performance

Links: # basic theory executive level knowledge, awareness, and control of cognitive activity in goal attainment; self-regulation processes in dynamic planning, monitoring, and adjustment of cognitive and task strategies; managing of affect/emotional control; persistence in the face of uncertainty, change, and difficulty; monitoring of teammates, work-load sharing; adaptation of individual and team performance # CTA requirements dynamic, emergent, ambiguous, information processing demands, shifting goal priorities/task strategies, requirements for team coordination and adaptive performance # scaled world requirements dynamic, emergent, ambiguous, information processing demands, shifting goal priorities/task strategies, requirements for team coordination and adaptive performance

Scenario Design and Measurement

Activities: # Translate critical CTA requirements to specific scenario features: dynamic, unfolding events; ambiguous cues for decision making; conflicting goals necessitate prioritization and trade-off strategies for individuals; shifting and unpredicatable overloads necessitate team cooperation and coordination of effort; dramatic increases in task difficulty and complexity require individual and team adaptation # Operationalize measures to capture critical psychological constructs, processes, and outcomes: cognitive ability, learning goal orientation, performance goal orientation, goal type, goal levels, goal commitment, feedback monitoring, task strategies, performance attribution, performance-goal discrepancies, self-efficacy, knowledge, performance, adaptive performance

Psychological Fidelity 38 Table 4 Illustration of Research Design Strategies

Basic Elements

Function

Theory of Instructional Design: # Integrated-Embedded Training Design - Self-regulation Theory # Team Compilation Theory - Self-regulation Theory

Guides the design of training manipulations to leverage key constructs/mechanisms specified by basic psychological theory. Two research strategies: # Research on Apure@ manipulations (isolated

constructs) designed to assess the impact on relevant self-regulation constructs and processes. Appropriate for basic research to evaluate basic theory or to assess new manipulation tools.

- mastery vs. performance goal frames; - feedback (individual, team, both) - team-level development vs. shifted individual to team-level development # Research on complex training strategies (a

combination of several Apure@ constructs) designed to optimize the instructional impacts on underlying processes. Appropriate for research to assess training and/or transfer effectiveness.

- sequenced mastery goals/feedback vs. sequenced performance goals/feedback

Theory of Training Effectiveness and Evaluation # Kraiger et al. (1993) # Kozlowski (1998) # Kozlowski & Salas (1997) # Kozlowski et al. (2001)

Guides research design features: # Assessment of multidimensional training outcomes to capture cognitive, affective, and behavioral domains. # Specifies developmental assessments at multiple time points during skill acquisition. # Specifies mulitlevel assessment -- individual and team -- in team training contexts. # Specifies assessment of skill generalization and adaptability.


Figure 1. Schematic Research Model

Research Factors:• pure constructs• training strategies

Individual Differences:• cognitive abilities• learning traits

TeamRegulation

IndividualRegulation

TeamOutcomes

IndividualOutcomes

TeamAdaptability

IndividualAdaptability

AntecedentsBasic Psychological

ProcessesTraining

EffectivenessTransfer and

Generalization

a psychological fidelity approach to simulation-based training - iacmr

Documents