is there a “fast-track” into the black box?: the cognitive models procedure robert r. hoffman,...
TRANSCRIPT
Is There a “Fast-track” Intothe Black Box?:
The Cognitive Models Procedure
Robert R. Hoffman, Ph.D.
John W. Coffey, Ed.D.
Mary Jo Carnot, M.A.
Institute for Human & Machine Cognition
University of West Florida
Poster presented at the 41st Annual Meeting
Psychonomics SocietyNew Orleans16 Nov 2000
Abstract
The Cognitive Models Procedure supports experts in crafting a model of their own reasoning, by relying on the presentation of "bogus" models. The goal is to have the expert re-construct a better model, which ideally will converge on the “modal model” of expert reasoning which includes the Duncker refinement cycle, recognition-priming, and situation awareness. The models are then cross-validated by: (1) field study of the observable behaviors that are entailed by models (e.g., the expert asserts that he always first inspects a certain data type), and (2). having a group of experts attempt to determine which of the models characterizes each of the experts in the team (or organization). This poster reports an application of this interview technique in a project on the reasoning and knowledge of expert weather
forecasters.
Application Domain
• Meteorology and Oceanography Training Facility, Pensacola Naval Air Station
• Pool of forecasters spanning Apprentice-Journeyman-Expert-Senior Expert levels of proficiency
Determination Standard Range Observed Range
LOW HIGHSenior Expert 50,000 - up 54,136 55,412Expert 40,000 - 49,999 47,064Junior Expert 30,000 - 39,999 31,032 36,898Senior Journeyman 25,000 - 29,999 25,720 29,127Journeyman 15,000 - 24,999 19,758 20,940Junior Journeyman 10,000 - 14,999 11,067 14,464SeniorApprentice 5,000 - 9, 999 6,384 8,448Junior Apprentice - 4,999 1, 848 2,016
Domain-appropriate Proficiency Scale
Knowledge Modeling
Utilized the following methods:– Structured Interviews– Protocol Analysis– Knowledge Audit– Critical Decision Method– Concept-Mapping
The Procedure
• Phase 1 - Choice among “bogus” models
• Phase 2 - Refinement
• Phase 3 - Guessing Game
• Phase 4 - Verification via direct observations
Base Model of Expertise
DataExamination
Action Queue
Course of action
Mental simulation,trend analysis, courses of action
Judgments,Predictions
Hypotheses,Questions
RecognitionPriming
Problem of the Day
Focus, leverage points,gaps, barriers
Mental Model
Refinement
Cycle
Action Plan
Refinement
Cycle
Situation
Awareness
Cycle
“Bogus” Models used in the CMP
MENTAL MODEL
COMPAREWITH MOS DISAGREE
INSPECT OTHERDATA
OUTPUTFORECAST
AGREE
REVISE
SURFACE DATA
UPPER AIR DATA, RADAR, GOES
MENTAL MODEL
COMPARE WITH MOS
REVISE
OUTPUTFORECAST
STYLE 1 STYLE 2
SURFACE DATA
Phase 1-2 Results
The strategy of providing the Participant with the "Bogus model" guidance in crafting a representation of their own reasoning strategy seems to have been successful. Each Participant balked at the bogus models, but then went on to craft a model that they felt comfortable with.
SURFACE OBSERVATIONS,SATELLITE IMAGERY
MENTALMODEL
OUTPUTFORECAST
EXAMINE FOR PERSISTENCE(past 24 hours ofobservations)
COMPARE WITHNUMERICALMODELS
ARE THERE ANY SALIENTDISAGREEMENTS IN THE DATA??
REVISE
DO ALL THE DATAAGREE?
A Journeyman's ModelExample Models
from Phase 1, 2
WATCHWEATHER CHANNEL
SKY WATCH
WALL OF THUNDER: SATELLITE IMAGES OBSERVATIONS (ASOS) LIGHTNING RADAR
MENTALMODEL
FORM
REFINE
NUMERICAL MODELS: NGM
COMPLETE DD-175-1 FORMS
NUMERICAL MODELS: ETA NOGAPS MM5 (4-DAY OUTLOOK)
PERSISTENCE( IN SUMMER)
LOCALKNOWLEDGE
PRODUCE FORECAST
EXAMINE WHEN TIMEPERMITS
EXAMINE
REFINE
EXAMINE
BEFORE ARRIVAL
AGREE
DISAGREE
REFINE
A Journeyman's Model
EXAMINE UPPER AIR DATA
COMPARE TO RADAR& SATELLITE DATA
FORM MENTAL MODEL
NGM
ETA
MM5
MOS
ACCOMMODATELOCAL EFFECTS
OUTPUT FORECAST
COMPARE TO NUMERICAL MODELS
AGREE
DISAGREE
DISAGREE
DISAGREEAGREE
AGREE
AGREE
REVISE
FORM REVISE
REFINEMENT CYCLE
REFINEMENT CYCLE
A SeniorExpert's Model
MENTALMODEL
SKY WATCHING
WALL OF THUNDER: ORDER IS OPPORTUNISTIC
FORM
REFINE
INCLUDES: HEURISTICS, CLIMATOLOGY, PERSISTENCE, PATTERN RECOGNITION
NUMERICAL MODELS:
REFINE IF EVIDENCE DISCONFIRMS
OUTPUT FORECAST
A Senior Expert's Model
Phase 3 Results
• Most participants adopted a "divide-and-conquer" strategy of first trying to identify models of senior experts or forecasters with whom they were more familiar, and identifying last the models that they thought were bogus and the models they thought were those of Apprentices.
• Of all of the identification judgments (N = 60), 13 or 22 percent were correct identifications.
Expert models were sometimes incorrectly identified as being models of Apprentices and bogus models.
Participants found the task to be an interesting challenge. Participants' comments during Phase 3 were revealing of the extent to which
they have opportunities to become familiar with one another's strategies. Participants may have opportunities to see what one another does, but do not
share much information about their actual strategies for data search, mental model formation and hypothesis testing.
Phase 4
• Observation of forecaster behavior when they first came on watch after a period of days when they had not been on the watchbill.
• Allowed probe questions: Understanding of the current weather situation? (e.g.,
"Is what you're seeing fit with persistence?" "Are the models agreeing?")
Skywatching (e.g. " Did you look at the Weather Channel before you came in?")
What are you going to do now? (e.g., "Are you going to look at the models?)
WALL OF THUNDER: SATELLITE IMAGES OBSERVATIONS (ASOS) LIGHTNING RADAR
MENTALMODEL
FORMREFINE
NUMERICAL MODELS: NGM
COMPLETE DD-175-1 FORMS
NUMERICAL MODELS: ETA NOGAPS MM5 (4-DAY OUTLOOK)
PERSISTENCE( IN SUMMER)
LOCALKNOWLEDGE
PRODUCE FORECAST
EXAMINE
REFINE
EXAMINE
WATCHWEATHER CHANNEL
SKY WATCH
BEFORE ARRIVAL
AGREE
DISAGREE
REFINE
DONE AT FDOWORKSTATION
AFFIRMED
AFFIRMED
AFFIRMED
NO NEED IN THISSITUATION(DONE LATER IN THEWATCH)
AFFIRMED
THIS WAS A PERSISTENCESITUATION
AFFIRMED
NOT NEEDED ON THISWATCH IN THISTIME PERIOD, BYHAPPENSTANCE
AFFIRMED
EXAMINE WHEN TIMEPERMITS
Example Phase 4 Results
MENTALMODEL
WATCHWEATHER CHANNEL
SKY WATCH
WALL OF THUNDER: ORDER IS OPPORTUNISTIC
NUMERICAL MODELS:
COMPLETE DD-175-1 FORMSAND OUTPUT FORECAST
LOCAL KNOWLEDGEHEURISTICS
FORM
MODIFY FORECASTS
UNFOLDING WEATHERESPECIALLY SEVERE
SITUATION AWARENESS CYCLE
REFINE
AFFIRMED
AFFIRMED
AFFIRMED
NOT AFFIRMEDIN THIS CASE (PERSISTENCE)
NOT NEEDED DURINGTHIS TIME PERIODON THIS WATCH
AFFIRMED
SURFACE OBSERVATIONS,SATELLITE IMAGERY
MENTALMODEL
OUTPUTFORECAST
EXAMINE FOR PERSISTENCE(past 24 hours ofobservations)
COMPARE WITHNUMERICALMODELS
ARE THERE ANY SALIENTDISAGREEMENTS IN THE DATA??
REVISE
DO ALL THE DATAAGREE?
GP DID NOT INSPECTSURFACE DATA
MENTAL MODELREVISION BASED ONSATELLITE IMAGERY
SKYWATCHING
THIS WAS NOT APERSISTENCESITUATION
RESOLVING HIS OWN MENTALMODEL WAS MORE IMPORTANT THANGETTING THE DATA INTO AGREEMENT
AFFIRMED
AFFIRMED
AFFIRMED
AFFIRMED
LEGEND SOLID BLACK = REVISED PATHSOUTLINE = COMMENTARIES
EVENT,QUERY, REPLY,ANALYSIS
TIME
E 4:40 P#2 arrived at METOC
E P#2 looked at the latest TAF
Q RH: "Did you skywatch on your way in? Did you look at theweather channel?"
R P#2: "Skywatch, yes. But we do not have a TV."
Q RH: "What was your understanding of the current weathersituation based on what you saw?"
R P#2: Thundering was somewhere.There was a cirrus mid-deck out to the north.There were cumulus towers--meaning instability--though theywere small.Must be decent capping as there was no thunder nearby.
E 4:40 P#2 looked at the Wall of Thunder--SIGMETS
E P#1 to P#2: "I don't know what's going on."P#2 to P#2: "You always know what's going on."P#1: "Everything's to the north of us."
E 4:46 P#2 inspects GOES water vapor loop on the FDO workstation.
E 4:51 P#1 told P#2 about the pilots that are out and en route back toNASP.
Example Results
Five elements of P#2's Model were affirmed:• Examination of satelli te imagery• Mental model formation• Mental model refinement• Comparison to computer model guidance• Reliance on local knowledge
Three elements of P#2's Model were qualif ied:• He did not begin by examining surface data.• This was not a persistence situation• Refinement focused on developing a coherent mental model
Four elements of the Base Model were affi rmed--• Data examination• Mental model formation• The refinement cycle• Reliance on local knowledge (i.e., this was not a persistence situation)
Participant Phase 1 TaskTime
Phase 3 TaskTime
Phase 4 TaskTime
P#1 23 minutes 20 minutes 21 minutesP#2 na 26 minutes 76 minutesP#3 15 minutes 10 minutes 30 minutesP#4 15 minutes 11 minutes 11 minutesP#6 9 minutes 17 minutes 42 minutesP#13 15 minutes naP#20 10 minutes 13 minutes
Total total 364 minutes
Conclusions
• Results from the procedure included models of the reasoning of seven forecasters, affirmed in a second phase, and then (for five of them), affirmed with modifications based on observations of actual forecasting behavior.
• For the total task time of 364 minutes, the procedure would average out to 52 minutes task time to develop a model. This figure is without doubt considerably less that than time takes in traditional experiments (i.e., think aloud problem solving with protocol analysis) to reveal and
verify reasoning models. • The CMP holds promise as a "fast track into the black box," allowing
the development of reasoning models and the testing of hypotheses concerning reasoning models in less time than taken by traditional experimentation.
References• Hoffman, R. R., & Markman, A. M. (Eds.) (in press). Human factors
in the interpretation of remote sensing imagery. NY: Lewis.• Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). A case study in
cognitive task analysis methodology: The Critical Decision Method for the elicitation of expert knowledge. Human Factors , 40 , 254-276.
• Hoffman, R. R., Shadbolt, N., Burton, A. M., & Klein, G. A. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129-158.
• Hoffman, R. R., Detweiler, M. A., Lipton, K., & Conway, J. A. (1993). Considerations in the use of color in meteorological displays. Weather and Forecasting, 8, 505-518.
• Hoffman, R. R. (1991). Human factors psychology in the support of forecasting: The design of advanced meteorological workstations. Weather and Forecasting, 6, 98-110.