scaled worlds: tractable, realistic, and engaging? wayne d. gray, ph.d. george mason university...
TRANSCRIPT
Scaled Worlds: Tractable, Realistic, and Engaging?
Wayne D. Gray, Ph.D.
George Mason University
Human Factors & Applied Cognition
[email protected] Tractable, Realistic, & Engaging? Slide 2
Motivation (1)
• Outstanding in My Field– Ft. Hunter-Ligget, NTC, Ft. Knox, Ft. Benning, Ft.
Hood, Schofield Barracks, W. Germany
• The Unfinished Revolution– Problem Finding (Newell & Simon, 1972)
• Protocol Analyses
• Real-World Experts
• Real-World Novices
– Unified Theories of Cognition (Newell, 1973)
[email protected] Tractable, Realistic, & Engaging? Slide 3
Motivation (2)
• Resistance from Academic Psychology?– Traditional Experimental Psychology slowing
yielding (work well-accepted by Cognitive Science Society types)
• Resistance from Human Factors Psychology?– Traditional HF Psychology -- Suspicious of
Theory– Ethnographic HF Psychology -- Suspicious of
Abstractions of the Task Environment
[email protected] Tractable, Realistic, & Engaging? Slide 4
Topic
• Simulations of the Task Environment
• How do various Task Environment Simulations differ?
• What inferences do different types of Task Environment Simulations permit us to draw?
[email protected] Tractable, Realistic, & Engaging? Slide 5
Outline
• The Problem
• Distinctions, Dimensions, Definitions
• Examples
• Scaled Worlds
[email protected] Tractable, Realistic, & Engaging? Slide 6
The Problem
In field research there is too much complexity to allow for definite conclusions; whereas, in laboratory research there is too little complexity to allow for interesting conclusions.
Brehmer, B. & Dörner, D. (1993). Experiments with computer-simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Computers in Human Behavior, 9(2-3), 171-184.
Three Dimensions of Simulated Task Environmentsdifficulttractablerealisticunrealisticengagingboring
Researcher
Real-World Task
Participant
[email protected] Tractable, Realistic, & Engaging? Slide 8
Definitions: Scaled World
• Beginning with a Complex, Real-World Task– A scaled world abstracts certain functional
relationships, while paring away others– Can be multiple scaled worlds of the same
complex, real-world task that differ on which functional relationships are abstracted and which are pared away
– The nature of the research question determines what is abstracted and what is pared away
[email protected] Tractable, Realistic, & Engaging? Slide 9
Definitions: Tractable
• The “complexity” issue– Can the researcher pursue the question of interest?
• Includes issues such as– Data collection (the right data, at the right grain size,
with the right timestamp)– Learnability issues– Usability issues– Computational cognitive modeling issues -- can the
model interact with the system?
[email protected] Tractable, Realistic, & Engaging? Slide 10
Definitions: Realism
• The simulated task environment is realistic to the situation to the degree to which experiences encountered in the environment will occur in the real-world
[email protected] Tractable, Realistic, & Engaging? Slide 11
Definitions: Engagement
• The simulated task environment is engaging to the degree to which it involves and affects the participants; that is, the degree to which they take it seriously
[email protected] Tractable, Realistic, & Engaging? Slide 12
Dimension difficulttractable
difficulttractableReal-WorldHi-Fidelity SimMicroworldSyntheticEnvironmentsScaled Worlds??????
Dimensions
Human OperatorTask in the Real-WorldSimulated Task EnvironmentEngagementextrinsic to the simulationknowledge possessed by the participantRealismrelevantirrelevantintrinsic to the task environment
unrealisticrealisticDimensionrealisticunrealisticReal-WorldHi-Fidelity SimMicroworldSyntheticEnvironmentsScaled Worlds??????
[email protected] Tractable, Realistic, & Engaging? Slide 15
Dimension boringengaging
engagingboringMicroworldSyntheticEnvironmentsReal-WorldScaled Worlds??????Hi-Fidelity Sim
[email protected] Tractable, Realistic, & Engaging? Slide 16
How Simulated Task Environments vary on Tractability, Realism and Engagement
• Four examples– Flight Simulator– VCR– Argus– Ned
• Each is followed by a discussion of whether it is or is not a scaled world
[email protected] Tractable, Realistic, & Engaging? Slide 17
Example: Flight Simulator
• As High Fidelity as $$$ and current technology can provide
• Is it a scaled world? NO
lowmedhigh¿¿engagement??realismtractability
[email protected] Tractable, Realistic, & Engaging? Slide 18
Example: VCR -- A High Fidelity Simulation of a Very Simple Task
[email protected] Tractable, Realistic, & Engaging? Slide 19
Gray, W. D. (in press). The nature and processing of errors in interactive behavior. Cognitive Science.
http://hfac.gmu.edu/Publications/pubs.html
lowmedhighengagement realismtractability
Is the VCR a Scaled World?
• No -- more like a high fidelity simulation of a low complexity task
[email protected] Tractable, Realistic, & Engaging? Slide 20
Example: Argus
[email protected] Tractable, Realistic, & Engaging? Slide 21
Is Argus a Scaled World?
• No -- Lacks the relation to a complex, real-world task that defines a scaled world
• It is a Synthetic Environment– Focus is on the study of interactive behavior under
conditions of high workload
lowmedhigh engagementrealismtractability
[email protected] Tractable, Realistic, & Engaging? Slide 22
Example: Ned
[email protected] Tractable, Realistic, & Engaging? Slide 23
Is Ned a Scaled World?
• Yes
lowmedhigh engagementrealismtractability
[email protected] Tractable, Realistic, & Engaging? Slide 24
Design Goals for Ned
• Has been carefully designed to abstract certain functional relationships of a complex, real-world task
• The functional relationships abstracted were defined by the goals of the study
• Most other functional relationships have been pared away
• Specifications for Ned arose from– Studying and listening to submarine Approach Officers
[email protected] Tractable, Realistic, & Engaging? Slide 25
Research Goals for Ned (1)
• Functional Relationships– Project goals entail the description and simulation of the
cognitive processes and memory structures used by Approach Officers in locating enemy submarines
• Impact on Cognitive Theory– To be able to say something new and different about the
expertise required for performing a complex, event-driven task
[email protected] Tractable, Realistic, & Engaging? Slide 26
Research Goals for Ned (2)
• Generalizability– Our goal is to be able to generalize the data and models of
Ned to a new, unknown submarine, with reduced manning, and new instrumentation
Caveat: “generalizability may be hindered by excessive attention
to mundane realism” (DiFonzo, Hantula, and Bordia 1998)
[email protected] Tractable, Realistic, & Engaging? Slide 27
Ned on the Tractability Dimension
• Ten displays with no overlapping information• Each object clicked is timestamped to nearest
tick (60hz) and saved to a log file• Researchers can “play back” Approach
Officer scenarios• Computational cognitive models written in
ACT-R can use Ned in same manner that our Approach Officers do.
[email protected] Tractable, Realistic, & Engaging? Slide 28
Ned on the Realism Dimension
• We decreased realism – by paring away the functional relationships provided
by the Approach Officer’s interactions with his crew and boat
• But, we maintained an information environment in which the functional relationships essential to our goals were preserved
• Used scenarios that represented situations familiar to the Approach Officer
[email protected] Tractable, Realistic, & Engaging? Slide 29
Ned on the Engagement Dimension
• Ned maintains a reasonable level of engagement– but only for those with much prior submarine
experience -- i.e., not suitable for college sophomores (or for this audience!!)
• The engagement arose from presenting Approach Officers with scenarios, information types, and displays that were recognizable as similar to those encountered in real missions
Dimensions Revisiteddifficulttractablerealisticunrealisticengagingboring
[email protected] Tractable, Realistic, & Engaging? Slide 31
Summary: Tractability
• Tractability– Microworld & Synthetic Environments ≥
Scaled World > High Fidelity Simulation– Tractability is defined by the needs of the
research question
[email protected] Tractable, Realistic, & Engaging? Slide 32
Summary: Realism
• Realism for scaled worlds– Hi-Fi simulation > Scaled World > Microworlds or
Synthetic Environments– The realism of scaled worlds is a focused subset of a
more complex, real-world task. – Focus is on a limited set of functional relationships
with a limited set contextual elements (e.g., scenarios, information types, and displays)
[email protected] Tractable, Realistic, & Engaging? Slide 33
Summary: Engagement
• Engagement for scaled worlds– Scaled worlds present schematics of the real world
situations– Participants supply the engagement via their
motivation and knowledge of the task and domain– Will not be more engaging than the real task