challenges in emergency and abnormal checklist design
TRANSCRIPT
Challenges in Emergency and Abnormal Checklist Design
Barbara Burian, Ph.D.San Jose State University Foundationat the NASA Ames Research Center
The Emergency and Abnormal Situations project is funded through the NASA Aviation Safety and Security Program.
Emergency and Abnormal Situations Project Industry Contacts and Consultants
Boeing, Bombardier, Airbus Industries, BAe Systems,
FAA, CAA (UK), JAA, ICAO,Eurocontrol, NavCanada
ALPA, APA, SWAPA, ATA, IATA, AFA, ADF
NTSB, TSB of Canada, ISASI
Airborne Express, Air Canada, Alaska, Aloha, American, Atlantic Southeast, Cathay Pacific, Continental, Delta, Fed Ex, Frontier, Hawaiian, Horizon, JetBlue, Southwest, United, UPS, US Airways, TWA (prior to merger)
Manufacturers:
Regulatory and Governmental Agencies:
Unions and Trade Groups:
Accident Investigation Bodies:
Airlines:
Emergency and Abnormal Situations Project
15 Different Categories of Issues – some are related to:
Emergency Equipment and Evacuation Issues
Automation and Automated Aircraft Systems
Training
Human Performance under Stress
Checklists and Procedures
Challenges in Emergency and Abnormal Checklist Design
Smoke, Fire, and Fumes
Checklists and Procedures
Smoke, Fire, and Fumes Checklists and Procedures
• Philosophies, company policies, and economic considerations
What Drives (or Should Drive*) the Design and Content?• Differences in aircraft and equipment design
• Type of operations – extended range, passenger vs. cargo
• History of the air carrier, History within the industry
• Assumptions about efficacy of crew response and expectations about amount of time available
• Human factors considerations, Understanding of human performancewhile under stress *
• Understanding of how different types of fires are ignited, fed, and spread
• Regulations, Advisory Circulars, etc.
Smoke, Fire, and Fumes Checklists and Procedures
Ambiguity of cues / level of certainty about situation
Conflicting warnings / cues
Smoke / Fumes of an unknown origin
Determining / Accessing the proper checklist
Length of time to complete procedures
Initiate descent / diversion and when
What type of descent profile
Checklist wording – how compulsory
Reduced visibility – font size, layout
What memory items
A Few of the Many Issues
Timing of source identification vs. smoke removal vs. descent initiation vs. fighting fire
High false smoke alarm rate
EROPS – nearest airport is far away
Ditching while on fire
How much troubleshooting
Fire in inaccessible places
Powering down electrical buses
Circuit breaker resetting
If / when to declare an emergency with ATC
Communicating / coordinating with Cabin crew
Smoke, Fire, and Fumes Checklists and Procedures
Ambiguity of cues / level of certainty about situation
Conflicting warnings / cues
Smoke / Fumes of an unknown origin
Determining / Accessing the proper checklist
Length of time to complete procedures
Initiate descent / diversion and when
What type of descent profile
Checklist wording – how compulsory
Reduced visibility – font size, layout
What memory items
A Few of the Many Issues
Timing of source identification vs. smoke removal vs. descent initiation vs. fighting fire
High false smoke alarm rate
EROPS – nearest airport is far away
Ditching while on fire
How much troubleshooting
Fire in inaccessible places
Powering down electrical buses
Circuit breaker resetting
If / when to declare an emergency with ATC
Communicating / coordinating with Cabin crew
Smoke, Fire, and Fumes Checklists and Procedures
Methods for Accessing the Correct Checklist:
• Gateway Checklist
• Several Separate Checklists
• One Integrated Checklist
What Drives (or Should Drive) the Design and Content?
• Philosophies, company policies, and economic considerations
Determining / Accessing the Proper Checklist
• Differences in aircraft and equipment design
• Type of operations – extended range, passenger vs. cargo
• History of the air carrier, History within the industry
• Assumptions about efficacy of crew response and expectations about amount of time available
• Human factors considerations, Understanding of human performancewhile under stress
• Understanding of how different types of fires are ignited, fed, and spread
• Regulations, Advisory Circulars, etc.
What Drives (or Should Drive) the Design and Content?
• Philosophies, company policies, and economic considerations
• Differences in aircraft and equipment design
• Type of operations – extended range, passenger vs. cargo
• History of the air carrier, History within the industry
• Assumptions about efficacy of crew response and expectations about amount of time available
• Human factors considerations, Understanding of human performancewhile under stress
• Understanding of how different types of fires are ignited, fed, and spread
• Regulations, Advisory Circulars, etc.
Determining / Accessing the Proper Checklist
Smoke, Fire, and Fumes Checklists and Procedures
Emergency Descent and Diversion Guidance:
Where in the Checklist and
How Stated?
If smoke/fumes are not eliminated, land at nearest suitable airport
Swissair 111 - In-flight Fire Nova Scotia, Canada September 2, 1998
Smoke, Fire, and Fumes Checklists and Procedures
Emergency Descent and Diversion Guidance Summary: Air Conditioning Smoke
1
2
1
2
Number of Checklists*
If smoke or fumes are persistent: Plan to land at the nearest suitable airport.Next to the last itemB767-400
N/ANowhereA320
Prepare to land immediately at nearest suitable airport.
Middle of checklist (step 7)CRJ-900
If smoke/fumes not eliminated, land at nearest suitable airport
Last item, 2nd
checklistMD-11
WordingLocationAircraft Type
(A Somewhat Unfair Comparison)
* to get to descent / diversion guidance
Smoke, Fire, and Fumes Checklists and Procedures
Emergency Descent and Diversion Guidance Summary: Electrical Smoke / Fire
1
N/AThere is no electrical smoke or fire checklist
Prepare to land immediately at nearest suitable airport.
First third of checklist (step 6)
1CRJ-900
1
-
Number of Checklists*
If smoke or fumes or fire persists or source is unknown: Plan to land at the nearest suitable airport.
Last item on the checklist
B767-400
LAND ASAP
Avionics Smoke has items for electrical smoke – 1st item
A320
WordingLocationAircraft Type
* to get to descent / diversion guidance
Smoke, Fire, and Fumes Checklists and Procedures
In a study of 15 in-flight fires that occurred between January 1967 and September 1998, the TSB of Canada determined that the average amount of time between the detection of an on-board fire and when the aircraft ditched, conducted a forced landing, or crashed was 17 minutes.
Emergency Descent and Diversion
Smoke, Fire, and Fumes Checklists and Procedures
Cost of Diversions: fuel, passenger ill-will, operational considerations,aircraft and crew scheduling, possible evacuationinjuries, etc.
D. Blake, 2000
Verified Smoke Events Versus Smoke Alarms
020406080
100120140160180
False Cargo Smoke Alarms, 1974 -1999
What Drives (or Should Drive) the Design and Content?
• Philosophies, company policies, and economic considerations
Descent and Diversion Guidance – Location and Wording• Differences in aircraft and equipment design
• Type of operations – extended range, passenger vs. cargo
• History of the air carrier, History within the industry
• Assumptions about efficacy of crew response and expectations about amount of time available
• Human factors considerations, Understanding of human performancewhile under stress
• Understanding of how different types of fires are ignited, fed, and spread
• Regulations, Advisory Circulars, etc.
What Drives (or Should Drive) the Design and Content?
• Philosophies, company policies, and economic considerations
Descent and Diversion Guidance – Location and Wording• Differences in aircraft and equipment design
• Type of operations – extended range, passenger vs. cargo
• History of the air carrier, History within the industry
• Assumptions about efficacy of crew response and expectations about amount of time available
• Human factors considerations, Understanding of human performancewhile under stress
• Understanding of how different types of fires are ignited, fed, and spread
• Regulations, Advisory Circulars, etc. (AC 120-80: “…flight crew members should begin planning for an emergency landing as soon as possible after the first indication of fire” pg 6.)
Immanuel Barshi, Ph.D.**Sean Belcher, M.A.**Ben Berman, A.B.**Barbara Burian, Ph.D.*Key Dismukes, Ph.D.**Richard Fariello, B.S.**
EAS Project Team
Colleen Geven, A.A.**Richard Geven, M.A.**Jon Holbrook, Ph.D.Jessica Lang Nowinski, Ph.D.Sharon Pickering, M.S.
* Certificated Pilot** ATP Certificate
Barbara Burian, Ph.D.
Emergency and Abnormal Situations Project
http://human-factors.arc.nasa.gov/eas
Checklist Design Factors (Checklists, Procedures, Manuals)
Physical Properties and Interface - size, weight, materials, hold in one hand?
Typography and use of Symbology - font, font size, boldface, intuitive symbology
Layout, Format, Organization - visual look, arrangement, “white” space
Purpose - fix, troubleshoot, manage situation, guide CRM
Objective (of checklist item) - direct action, inform, assess, make decision
Length - physical length, timing length
Content - Nomenclature & Abbrev. - terms, labels, abbreviations
Content – Language, Grammar, & - English?, verb tense, reading difficulty, clarity, Wording orientation/perspective, directiveness
Content - Level of Detail - amount of information provided
Content - Engineering Correctness - all necessary steps included
Content - Engineering Coherence - order of steps/timing makes “sense” to aircraft
Content - Logical Coherence - order of actions makes sense to the pilot and make “sense” operationally
Progression - movement within & between checklists/manuals
Access - finding correct checklist, prime real estate pgs.
Checklist Design Factors (Checklists, Procedures, Manuals)
Physical Properties and Interface - size, weight, materials, hold in one hand?
Typography and use of Symbology - font, font size, boldface, intuitive symbology
Layout, Format, Organization - visual look, arrangement, “white” space
Purpose - fix, troubleshoot, manage situation, guide CRM
Objective (of checklist item) - direct action, inform, assess, make decision
Length - physical length, timing length
Content - Nomenclature & Abbrev. - terms, labels, abbreviations
Content – Language, Grammar, & - English?, verb tense, reading difficulty, clarity, Wording orientation/perspective, directiveness
Content - Level of Detail - amount of information provided
Content - Engineering Correctness - all necessary steps included
Content - Engineering Coherence - order of steps/timing makes “sense” to aircraft
Content - Logical Coherence - order of actions makes sense to the pilot and make “sense” operationally
Progression - movement within & between checklists/manuals
Access - finding correct checklist, prime real estate pgs.
If Pack Fault due to low bleed air supply, thena bleed leak does not exist, and if WING ANTI-ICE is not required:
If Pack Fault due to low bleed air supply, and if a bleed leak does not exist, and if WING ANTI-ICE is not required:
Checklist Design Factors (Checklists, Procedures, Manuals)
Physical Properties and Interface - size, weight, materials, hold in one hand?
Typography and use of Symbology - font, font size, boldface, intuitive symbology
Layout, Format, Organization - visual look, arrangement, “white” space
Purpose - fix, troubleshoot, manage situation, guide CRM
Objective (of checklist item) - direct action, inform, assess, make decision
Length - physical length, timing length
Content - Nomenclature & Abbrev. - terms, labels, abbreviations
Content – Language, Grammar, & - English?, verb tense, reading difficulty, clarity, Wording orientation/perspective, directiveness
Content - Level of Detail - amount of information provided
Content - Engineering Correctness - all necessary steps included
Content - Engineering Coherence - order of steps/timing makes “sense” to aircraft
Content - Logical Coherence - order of actions makes sense to the pilot and make “sense” operationally
Progression - movement within & between checklists/manuals
Access - finding correct checklist, prime real estate pgs.
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
1
The Limits of Expertise:The Misunderstood Role of
Pilot Error in Airline Accidents
Key DismukesNASA Ames Research Center
Ben Berman and Loukia LoukopoulosSan Jose State University Foundation at
NASA Ames Research Center
ALPA Air Safety Week19 August 2004
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
2
Approach
• Reviewed NTSB reports of the 19 U.S. airline accidents between 1990-2000 attributed primarily to crew error.
• Asked: Why might any airline crew in situation of accident crew be vulnerable to the same error?
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
3
Hindsight Bias
• Knowing the outcome of an accident flight reveals what crew should have done differently.
• Accident crew does not know the outcome.– They respond to situation as they perceive it at the
moment.
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
4
Immediate demands of situation & tasks
being performed
Training, experience,& personal goals
Social/OrganizationalInfluences- Formal procedures &
policies- Explicit goals & rewards- Implicit goals & rewards
Actual “norms” for line operations
Human cognitioncharacteristics &
limitations
Crew responses to situation
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
5
A Truism
• No one thing “causes” accidents.
• Confluence of multiple events, actions taken or not taken, and environmental factors.
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
6
Confluence of Factors in a CFIT Accident
Non-precision approach ≥ 250 foot
terrain clearance
Are most pilots aware of this?
Weather conditions
Airline’s use of QFE altimetry
Strong crosswind
Autopilot would not hold
PF selected Heading Select
Additional workload
Increased vulnerabilityto error
Crew error (70 feet) in altimeter setting
Altimeter updatenot available
170 foot error in altimeter reading
Tower closed
Tower windowbroke
Rapid change in barometric pressure
Approach controllerfailed to updatealtimeter setting
Altitude Hold may allow altitudesag 130 feet in
turbulence
PF used Altitude Hold to capture MDA
PM used non-standard callouts to alert PF
Training & Standardization issues?
Aircraft struck trees310 feet below MDA
?
?
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
7
Chance Combination of Contributing Factors
• Airline accidents are extremely rare in modern operations– Countermeasures in place for single-point failures of equipment
or human performance
• Occasionally accidents slip through multiple defenses because of chance combination of multiple factors– Number of possible combinations/permutations of factors is vast– Difficult to devise countermeasures
• Vague advice to pilots to “break the accident chain” is not helpful
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
8
Six Overlapping Clusters of Error Patterns
1) Inadvertent slips and oversights while performing highly practiced tasks under normal conditions
2) Inadvertent slips and oversights while performing highly practiced tasks under challenging conditions
3) Inadequate execution of non-normal procedures under challenging conditions
4) Inadequate response to rare situations for which pilots are not trained
5) Judgment in ambiguous situations
6) Deviation from explicit guidance or SOP
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
9
1) Inadvertent Slips/Oversights in Practiced Tasks under Normal Conditions
• Examples: – Omitting procedural step or checklist item– Remembering altimeter setting incorrectly– Misjudging landing flare
• Identical to errors pilots frequently report to ASRS and ASAP and errors observed in LOSA
• Commonplace error had to combine with multiple other factors to result in accident
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
10
2) Inadvertent Slips/Oversights in Practiced Tasks under Challenging Conditions
• Probability of commonplace errors goes up with workload, time pressure, fatigue and stress
• Snowball effects: events/decisions/actions increase workload, time pressure, and stress downstream, increasing chance of more problems and errors
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
11
3) Inadequate Execution of Non-normal Procedures under Challenging Conditions
• Failure to recover from spiral dive, stall, or windshear
• Veridian study suggests existing training not sufficient
• Surprise, confusion, and stress may impede correct diagnosis of upset and timely execution of appropriate procedure
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
12
4) Inadequate Response to Rare Situations for which Pilots are not Trained
• Examples: – False stickshaker activation just after rotation– Oversensitive autopilot drove aircraft down at Decision
Height– Anomalous airspeed indications past rotation speed– Uncommanded autothrottle disconnect with non-salient
annunciation
• Surprise, confusion, stress, and time pressure play a role
• No data on what percentage of airline pilots would respond adequately in these situations
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
13
5) Judgment and Decision-making in Ambiguous Situations
• Examples:– Continuing approach in vicinity of thunderstorms– Not de-icing or not repeating de-icing
• No algorithm to calculate when to break off approach; company guidance usually generic
• Crew must integrate incomplete and fragmentary information and make best judgment
– If guess wrong, crew error is found to be “cause”
• Accident crew judgment & decision-making may not differ from non-accident crews in similar situations:
– Lincoln Lab study: Penetration of storm cells on approach not uncommon– Other flights may have landed or taken off without difficulty a minute or two
before accident flight
• Questions:– What are actual industry norms for these operations?– Sufficient guidance for crews to balance competing goals?– Implicitly tolerate/encourage less conservative behavior as long as crews
get by with it?
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
14
6) Deviation from Explicit Guidance or SOP
• Example: Attempting to land fromunstabilized approach resulting from slam-dunk approach
• Simple willful violation or more complex issue?– Are stabilized approach criteria published/trained as
guidance or absolute bottom lines?– What are norms in company and the industry?
• Pilots may not realize that struggling to stabilize approach before touchdown imposes such workload that they cannot evaluate whether landing will work out
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
15
Cross-Cutting Factors Contributing to Crew Errors
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
16
Situations Requiring Rapid Response
• Nearly 2/3 of 19 accidents• Examples: upset attitudes, false stickshaker
activation after rotation, anomalous airspeed indications at rotation, pilot-induced oscillation during flare, autopilot induced oscillation at Decision Height
• Very rare occurrences, but high risk• Surprise is a factor• Inadequate time to think though situation
– automatic response required
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
17
Challenges of Managing Concurrent Tasks
• A factor in great majority of accidents
• Workload quite high in some accidents but in most time was available to perform all tasks
• Especially vulnerable to error when switching attention among tasks, interrupted, distracted, or forced to defer tasks out of normal sequence
• Vulnerability inherent in basic cognitive processes:– Can attend to only one distinct task at a given instant– Once attention is diverted from a task do not always remember
to resume task if not prompted
• Better monitoring can help prevent/catch errors– But monitoring is itself a concurrent task and vulnerable to the
same factors that produce errors
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
18
Equipment Failures and Design Flaws
• Occurred in 2/3 of these accidents
• Some equipment failures/design flaws precipitated chain of eventsExample: false stickshaker after rotation
• Some equipment failures/design flaws undermined efforts of crew to respondExample: stickshaker failed to activate when aircraft
approached stall
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
19
Stress
• Hard to evaluate extent, but stress is normal physiological/behavioral response to threat
• Acute stress hampers performance– Narrows attention (“tunneling”)– Reduces working memory capacity
• Combination of surprise, stress, time pressure, and concurrent task demands can be lethal setup
• Beginning NASA research project on effects of stress on crew performance
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
20
Shortcomings in Training and/or Guidance
• >1/3 of accidents
• Inadequate guidance to pilots about known problems (e.g. high sensitivity of wings without leading edge devices to minute amounts of frost)
• Upset attitude recovery training
• How to deal with fact that not possible to train for every possible situation?
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
21
Plan Continuation Bias
• Unconscious cognitive bias to continue original plan in spite of changing conditions
• Appears stronger as one nears completion of activity (e.g., approach to landing)
• Bias may prevent noticing subtle cues indicating original conditions have changed
• May combine with other cognitive biases:– frequency sampling bias (“It’s always worked before”)– reactive responding is easier than proactive thinking
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
22
Social/Organizational Issues
• Actual norms may deviate from Flight Operating Manual – Little data available on extent to which accident
crews actions are typical/atypical
• Little study or acknowledgment of competing pressures– e.g., on-time performance vs. conservative
response to ambiguous situations– Pilots may not be consciously aware of influence
of internalized competing goals
Click to edit Master title style
• Click to edit Master text styles• Second level• Third level• Fourth level• Fifth level
23
Countermeasures? No Easy Solutions• Recognize that most accidents are systems accidents
– Identify hidden vulnerabilities of systems– Pilots, managers, designers of equipment & procedures should be well
educated about human cognitive characteristics & limitations– Review procedures to insure they do not contribute to inherent cognitive
vulnerabilities– Insist on conservative hard-bottom lines in critical situations
• Need better info on how airspace system typically operates and how crews respond– LOSA and FOQA
• Beef up training– upset attitude recovery and other rapid response situations– monitoring
• Acknowledge inherent trade-offs between safety and system efficiency– Include all parties in analysis of trade-offs– Make policy decisions explicit