improving cockpit task management performance:
DESCRIPTION
Aviate. Navigate. Communicate. Manage Systems. Improving Cockpit Task Management Performance:. The AgendaManager Training Pilots to Prioritize Tasks. Observation: Cockpit Task Management Errors. Cockpit (flight deck) is a multitask environment aviate navigate communicate manage systems - PowerPoint PPT PresentationTRANSCRIPT
Improving Cockpit Task Management Performance:
Aviate
Navigate
Communicate
Manage Systems
The AgendaManagerTraining Pilots to Prioritize Tasks
Aviate
Navigate
Communicate
Manage Systems
CTM
2
Observation: Cockpit Task Management Errors
• Cockpit (flight deck) is a multitask environment– aviate– navigate– communicate– manage systems
• Results of distraction, preoccupation– Everglades L-1011 accident– many incidents
• Hypotheses: – flightcrew must manage as well as perform tasks:
Cockpit Task Management (CTM) – CTM is a significant factor in flight safety
Aviate
Navigate
Communicate
Manage Systems
CTM
3
Preliminary Normative Theory of CTM
• initiate tasks to achieve goals
• assess status of all tasks
• terminate completed and ‘obsolete’ tasks
• prioritize remaining tasks based on– importance:
• aviate
• navigate
• communicate
• manage systems
– urgency
– other factors (?)
• allocate resources (attend) to tasks in order of priority
Aviate
Navigate
Communicate
Manage Systems
CTM
4
Cockpit Task Management Research
• CTM Errors in Aircraft Accidents (1991)– 80 CTM errors in 76 (23%) of 324 accidents
• CTM Errors in Critical, In-Flight Incidents (1993)– 349 CTM errors in 231 (49%) of 470 incident reports
• Part-Task Flight Simulator Study (1996)– CTM error rate increases with workload
• ASRS Study of CTM and Automation (1998)– Task prioritization error rate higher in advanced technology reports
• Findings:– CTM is a significant factor in flight safety
– CTM can potentially be improved
Aviate
Navigate
Communicate
Manage Systems
CTM
Improving CTM Through Technology:
The AgendaManager
5
Aviate
Navigate
Communicate
Manage Systems
CTM
6
Statement of Needs and Requirements Definition
• CTM aid shall– maintain a current model of aircraft state and current
cockpit tasks,
– monitor task state and status,
– compute task priority,
– remind the flightcrew of all tasks that should be in progress, and
– suggest that the flightcrew attend to tasks that do not show satisfactory progress.
– leave the pilot in control
Aviate
Navigate
Communicate
Manage Systems
CTM
7
System Analysis
• Generic, twin-engine transport aircraft– major subsystems
• power plant
• fuel system
• electrical system
• hydraulic system
• adverse weather system
• autoflight system
• flight management system.
– state variables of importance to pilot
specifications for simulator
Aviate
Navigate
Communicate
Manage Systems
CTM
8
Basic and Detailed Design of
The AgendaManager• Object-Oriented Design
– things & activities from IDEF0 models objects
• Multi-Agent Approach– AMgt functions are complex, cognitive functions AI– AMgt is complex interplay of many entities DAI
• System Agents• Actor Agents• Goal Agents• Function Agents• Agenda Agent• Agenda Manager Interface
• Display Design– general display design guidelines alternative display designs– consistency with EICAS final display design
Autoflight
L Engine
Hyd System
Fuel System
Aircraft
Pilot
Aircraft Agent
Autoflight Agent
L Engine Agent
Hyd System Agent
Fuel System Agent
Flightcrew Agent
Simulator
AMgr display
Verbex ASR
AgendaManager
descend to 9,000 ft G & F Agents
descend to 8,000 ft G & F Agents
extinguish L ENGINE FIRE G & F Agents
restore C HYD PRESS G & F Agents
correct FUEL BALANCE G & F Agents
Goal & Function AgentsSystem AgentsSystem Models
reduce to 240 kt G & F Agents
maintain 070 deg G & F Agents
maintain 070 deg G & F Agents
reduce to 240 kt Goal & Function Agents
information flow
unsatisfactory functionsconflicting goals
satisfactory functions
AMgr Architecture and Function
Aviate
Navigate
Communicate
Manage Systems
CTM
10
Simulator(with EICAS)
Aviate
Navigate
Communicate
Manage Systems
CTM
11
AMgr Display(replaced EICAS)
Aviate
Navigate
Communicate
Manage Systems
CTM
12
AMgr Operation
• simulator runs• pilot declares goals via ATC acknowledgements• System & Actor Agents instantiate Goal Agents• Goal Agents watch for goal conflicts• Function Agents assess function status• AgendaManager informs pilot via display
extinguish L engine fire not OK -> continuing
slow to 240 kt fast -> acceleratingmaintain 070 degdescend to 7,000 ft A/F alt goal conflict
correct fuel balance L heavy -> increasing
AgendaManager Display Design
extinguish L engine fire not OK -> continuing
slow to 240 kt fast -> acceleratingmaintain 070 degdescend to 7,000 ft A/F alt goal conflict
correct fuel balance L heavy -> increasing
extremely important, urgent goals
(highest priority) trend info
aviate goals(high priority)
system goals(lower priority)
gray = OKamber = not OKred = important/urgent not OK
maintain 280 ktmaintain 120 degmaintain15,000 ft
Initial Conditions:altitude = 15,000 ftheading = 120 degspeed = 280 ktall systems normal
maintain 280 ktmaintain 120 degdescend to 11,000 ft high -> descending
ATC: “... descend and maintain 11,000 ft”pilot: “Roger, “... descend and maintain 11,000 ft”
sets A/F altitude to 11,000 ftdescent begins
maintain 280 ktturn L to 070 deg right of -> turning Lmaintain 11,000 ft
ATC: “... turn left heading 070”pilot: “Roger, “... turn left heading 070”
begins turnlevels off at 11,000 ft
maintain 280 ktmaintain 070 degmaintain 11,000 ft
correct fuel balance L heavy -> unbalancing
pilot: rolls out on 070 degAMgr:detects fuel imbalance & displays it
slow to 240 kt fast -> slowingmaintain 070 degdescend to 9,000 ft high -> descending
correct fuel balance L heavy -> balancing
pilot: begins fuel crossfeedATC: “... descend and maintain 9,000 ft; reduce
speed to 240 kt”pilot: “Roger ... descend and maintain 9,000 ft;
reduce speed to 240 kt”sets altitude to 9,000 ft, descent beginsreduces throttles, aircraft slows
extinguish L engine fire not OK -> continuing
slow to 240 kt fast -> acceleratingmaintain 070 degdescend to 7,000 ft A/F alt goal conflict
correct fuel balance L heavy -> balancing
AMgr: detects left engine firepilot: “... we have a problem ...”ATC: “... descend and maintain 7,000 ft”pilot: “Roger ... descend and maintain 7,000 ft”
mis-sets altitude to 6,000 ftspeed increases
maintain 240 ktmaintain 070 degmaintain 7,000 ft
correct fuel balance R heavy -> unbalancing
fire outspeed controlledpilot: sets A/F to 7,000 ft
forgets to secure crossfeed when fuel balanced
Aviate
Navigate
Communicate
Manage Systems
CTM
22
Test and Evaluation (1)
• Objective: compare AMgt performance (AMgr vs EICAS)
• Apparatus– flight simulator
– AMgr
• Subjects: 8 line pilots
• Scenarios:– EUG to PDX
– PDX to Eugene
• Primary factor: monitoring and alerting condition– AMgr
– EICAS
Aviate
Navigate
Communicate
Manage Systems
CTM
23
Test and Evaluation (2)
• General Procedure– subject introduction– automatic Speech Recognition system training– flight training (using MCP)– subsystem training (fault correction)
– EICAS/AMgr training
• Trials– Scenario 1 (EICAS/AMgr)
• experimenter/ATC controller gives clearances, induces faults, induces goal conflicts
• subject acknowledges clearances, flies simulator, corrects faults, detects and resolves goal conflicts
– Scenario 2 (AMgr/EICAS)
Aviate
Navigate
Communicate
Manage Systems
CTM
24
Response variable AMgr EICAS sig. levelwithin subsystem correct prioritization 100% 100% NS
susbsystem fault correction time (sec) 19.5 19.6 NS
autoflight system programming time (sec) 7.0 5.9 NS
goal conflicts corrected percentage 100% 70% 0.10
goal conflict resolution time (sec) 34.7 53.6 0.10
subsystem/aviate correct prioritization 72% 46% 0.05
average number of unsatisfactory functions 0.64 0.85 0.05
percentage of time all functions satisfactory 65% 52% 0.05
mean subject effectiveness rating (-5 to 5) 4.8 2.5 0.05
Evaluation Results
Aviate
Navigate
Communicate
Manage Systems
CTM
25
Conclusions
• CTM is a significant factor in flight safety.• CTM can be facilitated (e.g., AMgr).• Future success of knowledge-based avionics depends
on a systematic approach to development:– systematic identification of problems, needs, and
opportunities
– appropriate application of appropriate technology
– evaluation of systems based on operationally relevant performance measures
Aviate
Navigate
Communicate
Manage Systems
CTM
Improving CTM Through Training:
Training Pilots to Prioritize Tasks
26
Aviate
Navigate
Communicate
Manage Systems
CTM
ResearchMotivation and Objective
• Is task prioritization trainable?• Evidence suggests that voluntary control of
attention is a trainable skill– e.g., Gopher (1992)
• Objective– Develop and evaluate a CTM training program to
improve task prioritization performance.
Aviate
Navigate
Communicate
Manage Systems
CTMMethodology
• Participants– 12 General Aviation pilots, IFR rated, with at least 100 hrs “pilot-
in-command” total time.
– Recruited through flyers and word of mouth
– Oregon State (Corvallis, Albany, Salem, Eugene, Portland)
• Apparatus: Microsoft Flight Simulator 2000– 3 monitors, Flight Yoke, Throttles, and Rudder Pedals
– IFR conditions
– Two flight scenarios
Aviate
Navigate
Communicate
Manage Systems
CTM
Lab Setup
Aviate
Navigate
Communicate
Manage Systems
CTM
Participant Display(C-182RG)
Aviate
Navigate
Communicate
Manage Systems
CTM
Experimenter’s Display
Aviate
Navigate
Communicate
Manage Systems
CTM
Experimental Groups
• Control Group: No Training• Descriptive Group: CTM lecture
• Multi-tasking• Attention• CTM• Task Prioritization errors• Accident/Incident examples• What to be aware of.
• Prescriptive Group:– CTM lecture – “APE” procedure
Aviate
Navigate
Communicate
Manage Systems
CTM
APE:Assess Prioritize Execute
• Let the APE help you– Assess the situation:
• aircraft systems, environment, tasks, procedures• “What’s going on?” “What should I be doing?”
– Prioritize your tasks:1. Aviate: “Is my aircraft in control?”2. Navigate: “Do I know where I am and where I’m going?”3. Communicate: “Have I communicated or received important information?”4. Manage systems: “Are my systems okay?”
– Execute the high priority tasks Now.
• Invoke the APE frequently.• Think out loud.
A P E
Aviate
Navigate
Communicate
Manage Systems
CTM
Experimental Procedure
1. Initial briefing, informed consent
2. Initial 30-minute simulator training
3. Pre-training flight
4. CTM training (break for control group)
5. Additional 30-minute simulator training
6. Post-training flight (different scenario)
7. Post-experiment questionnaire
Aviate
Navigate
Communicate
Manage Systems
CTM
Dependent Measures
• Task prioritization error rate– 19 Task prioritization challenges, e.g.
• clearance near end of climb
• “bust” altitude? (+/- 200 ft)
• Prospective memory recall rate– 5 Memory recall challenges (prospective memory), e.g.,
• “report crossing SHONE [intersection]”
• remember to report?
Aviate
Navigate
Communicate
Manage Systems
CTM
Data Collection
• Flight Data Recorder• Videotape• Observation• Data reduction to:
– task prioritization error rate
– prospective memory recall rate
Aviate
Navigate
Communicate
Manage Systems
CTM
Results: ANOVA(task prioritization error rate)
Effect df SS df MS F-ratio p-value
Group 2 .1914125 9 .0799194 2.395 .147
Flight 1 .2053500 9 .0088806 23.123 .001
Group x Flight 2 .0429125 9 .0088806 4.832 .038
Aviate
Navigate
Communicate
Manage Systems
CTM
Interaction Plot(task prioritization error rate)
Interaction Plot
Flight
Err
or R
ate
GroupControlDescriptivePrescriptive
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Pre Training Post Training
Control
Descriptive
Prescriptive
Aviate
Navigate
Communicate
Manage Systems
CTM
Results: ANOVA(prospective memory recall rate)
Effect df SS df MS F-ratio p-value
Group 2 .017 9 .028 .603 .568
Flight 1 .074 9 .034 2.181 .174
Group x Flight 2 .171 9 .034 5.055 .034
Aviate
Navigate
Communicate
Manage Systems
CTM
Interaction Plot(prospective memory recall rate)
Interaction Plot
Flight
Mem
ory
Tas
ks
GroupControlDescriptivePrescriptive
0.5
0.6
0.7
0.8
0.9
1
Pre Training Post Training
Prescriptive
Descriptive
Control
Aviate
Navigate
Communicate
Manage Systems
CTM
Paired t-tests
• Prescriptive training group improved• Task prioritization error rate
• Prospective memory recall rate
• Descriptive training group improved• Task prioritization error rate
• Control group did not significantly improve
Aviate
Navigate
Communicate
Manage Systems
CTM
Discussion
• Task Prioritization Error rate– Reduced, perhaps, due to (Prescriptive) CTM training.– Significant interaction and post-hoc tests support that
hypothesis.
• Prospective Memory Recall rate– Increased, perhaps, due to (Descriptive & Prescriptive) CTM
training.– Significant interaction and post-hoc tests support that
hypothesis.
Aviate
Navigate
Communicate
Manage Systems
CTM
Possible Interpretations
• Results may have two interpretations:1. CTM training did improve task prioritization performance.
2. CTM training did not improve task prioritization.• Floor effect
• MSFS experience
• Age
• Research favors first interpretation• ANOVA results
• t-tests
• Potential for better control group performance was there.
• Additional tests
Aviate
Navigate
Communicate
Manage Systems
CTM
Final Comments
• CTM performance significant to flight safety• Results are encouraging• Evidence suggests that task prioritization is a
trainable skill• Follow-up experiment underway to resolve
ambiguities• If successful, would provide evidence that CTM
training can reduce risk of CTM errors and subsequent accidents
Aviate
Navigate
Communicate
Manage Systems
CTM
The AMgr: a KBS 45
The Cockpit Task Management Website
http://flightdeck.ie.orst.edu/CTM/