accident investigation techniques and methodologies chuck dejohn, d.o., m.p.h federal aviation...

Accident Investigation Techniques and Methodologies

Chuck DeJohn, D.O., M.P.HFederal Aviation AdministrationCivil Aerospace Medical Institute

http://www.iprr.org/Papers/Defectslides/sld004.htm

Frei R, Kingston J, Koornneef F, & Schallier P. Investigation Tools in Context. JRC European Commission Institute for Energy Seminar. Investigation of Accidents. May 2003. Petten Netherlands.

Methods vs. MethodologyMethodology: A system of principles, practices and

body of procedures (methods) applied to a specific branch of knowledge. An overall approach to a field such as accident investigation. Examples: Adversarial, Commission, Events

Reconstruction, Modeling, Simulation

Method: A technique or tool. A regular, disciplined, systematic set of procedures used according to an underlying, detailed, logically ordered plan. Examples: Multi-linear Events Sequencing (MES), Fault

Tree Analysis (FTA), Management Oversight and Risk Tree (MORT)

Benner, L. Methodology biases which undermine accident investigation. Proceedings of ISASI 18th Annual Seminar. Washington, DC. 1981.

Methods vs. MethodologyProblems

Defined differently: By different authors By the same author in different articles By the same author in the same article!

Methods and Methodologies are often used interchangeably Examples include Fault Tree Analysis (FTA) and

Management Oversight and Risk Analysis Tree (MORT)

Benner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.

Methodology Classification Schemes

Unstructured Adversarial Events

Reconstruction Modeling Simulation

SurveyArchivalHistoricalExperimentalCase Study

vs.

Common SenseAdversarialEngineeringStatisticalSymbolic Modeling

vs.

Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers.

Benner L. Methodological biases which undermine accident investigations. 1981. Proceedings of the 18th Annual ISASI Seminar. Washington, D.C.

Unstructured Methodology

“Common Sense” or “Hunt-and-Peck”“Who, what, when, where, how and why?” Sequential ordering of eventsExplanation of the accident is acceptable if it

“makes sense”Truth is determined by the investigator

Benner, L. Methodology biases which undermine accident investigation. Proceedings of ISASI 18th Annual Seminar. Washington, DC. 1981

Adversarial MethodologyRules of evidence and judicial procedures Opposing interests will bring out the truth

Facts are gathered by the parties and informally tested by discussion against hypothesis for logic and consistency

Reasoned conclusions logically drawn from technical evidence

ExamplesU.S. Party System used by the NTSBCommission inquires


Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers ex libris.

Events Reconstruction Methodology

Reconstruction of sequence of events (SOE): Physical evidence Witness interviews Speculation by investigator

Methodology is not rigorous “Events” are undefined and highly variable Logic trees often culminate in event(s) selected by

investigator without showing time relationshipsProbable cause (PC) often selected from one or more of the

events


Symbolic Modeling Methodologies

Pictorial representations of the SOEFault Trees

Failure selected and all possible factors that can contribute to the event are diagramed in the form of a tree

Not always considered an overall methodologyExamples: Logic Tree Analysis, Fault Tree Analysis

(FTA), Management Oversight and Risk Tree (MORT), Multilinear Events Sequencing (MES)

Ferry TS. Modern accident investigation and analysis. Pp 134-44. 1981. John Wiley & Sons. New York.

Harvey MD. Models for accident investigation. 1985. Alberta Occupational Health and Safey Division. Occupational Health and Safey Division.


EM 1110-2-6050. 30 Jun 99. Appendix F. Use of Logic Trees in Probabilistic Seismic Hazard Analysis.

Simulation Methodologies

Reenactments that allow investigators to vary assumed events and asses effects of changesFormulate hypothesesDevelop data where there are gapsExamples:

Computerized modelingScale modelingUse of actual aircraft/systems

Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers ex libris

Methodology Rankings

Compare simultaneous investigations of the same accident using different methodologiesVery resource intensive

1985 Benner Study:17 U.S. Federal Government Agencies10 evaluation criteria

Benner L. Investigating investigation methodologies. Starline Software Ltd. Oakton, VA. 2003. http://members.cox.net/lbjr99/papersa/IRIA03bennerf.pdf on 5/20/04.


Methodology RankingsAgencies Studied

Consumer Product Safety Commission

Department of Agriculture

Department of the Air Force

Department of the Army

Department of Energy

Department of Labor Mine Safety and Health Administration

Department of Labor Occupational Safety and Health Administration

US Coast Guard

Federal Highway Administration

National Highway Traffic Safety Administration

General Services Administration

Library of Congress

National Aeronautics and Space Administration

National Institute of Occupational Safety and Health

National Transportation Safety Board

Navy Department

Nuclear Regulatory CommissionBenner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.

Methodology RankingsRating Criteria

Encouragement: Encourages harmonious participation.

Independence: Produces unimpeachable results.

Initiatives: Supports personal initiative.

Discovery: Supports timely discovery of facts.

Competence: Provides/improves employee competence.

Standards: Provides for review of safety and health standards.

Enforcement: Supports the enforcement program.

States: Encourages states to take responsibility.

Accuracy: Outputs can be tested for completeness, validity, logic and relevance.

Closed Loop: Compatible with pre-investigation outputs.


Methodology RankingsTop Three

Event reconstructionModeling

MORTFault Tree

Adversarial


Accident Investigation MethodsMethods are Tools used by the investigator, not

an overall system or branch of knowledge Most methods are sequencing tools – Reduce

accidents to a collection of events using cause and effect relationships

Fault Tree Analysis (FTA)Management Oversight and Risk Tree Analysis (MORT)Multilinear Events Sequencing (MES)Sequentially Timed Events Plotting (STEP) Events and Causal Factors Analysis (ECFA)Root Cause Analysis (RCA)


Accident Investigation Methods

To select the best method you should know:The name of the method you use nowWhich methods are availableWhich methods are better than othersThe outputs of the method you chose

FTA Created at Bell Laboratories, refined by Boeing to

analyze Minuteman missile problems and later adopted by DOD.

Selected failure and all possible factors that can contribute are diagrammed into a tree. The accident is the “top event.”

Top-down approach to determine how “top events” can be caused by individual or combined lower level failures.

Events – Failures that lead to accidents. Gates – Ways failures combine to cause accidents.

Useful for large accident investigations.

Ferry TS. Modern accident investigation and analysis. Pp. 134-44. 1981. John Wiley & Sons. New York.

Schiodtz K. Fault tree analysis in the application of accident analysis. 2003.

FTA

AdvantageConveniently represents main causes/factors

of an accident

DisadvantagesNo temporal relationships between events No ordering of events“Actors” not shown


FTA of Aircraft Runway Overrun Accident

Erickson, C.A. Accident Investigation Using EEFTA. Proceedings of the 18th International System Safety Conference. Seattle, Washington. 2000.

MORT

Developed in the 1960s in response to the lack of accident investigation techniques that existed to support rigorous analysis

Pre-designed, systematized logic tree in a generic graphical checklist format of approximately 1500 items

Best suited to large complex accident Requires extensive training


American Society of Safety Engineers. Northern Illinois University. TECH 438. MORT, Mini-MORT & PET. 2003.

Mort Event Symbols


Mort Logic Gates


MORT Advantages

Systematically examines all possible causal factors Ideal when there is a shortage of expertise to ask the right questions Evaluates multiple causes

Works well for complex accidents involving multiple systemsAddresses root causes and contributory causes

Looks beyond immediate causes including management/program factors

Disadvantages Time consuming and tedious to use

Requires extensive training Inappropriate for relatively simple accidents

Can focus more on management than the accident event May lead to recommendations that are too broad (i.e. more training,

more supervision) No temporal relationships between events

Department of Energy. Accident Investigation Program. Section 7 – Analyzing Data. Oct 1999.

Department of Energy Accident Investigation Program. Root cause analysis. January 19, 2001.

Abbreviated MORT Diagram

LTA implies Less Than Adequate performance

PG Bishop, et al. Learning from incidents involving E/E/PE systems. Part 1. 2003. HSE Books. Norwich.

MESTime line chart of the accident process:Time

line is displayed at the bottom of the chart and conditions and events are shown in logical order.

Event = Actor + ActionEvent: Something of significance caused by an action.

Actor: One who causes an event to occur. Does not have to be a person.

Action: Acts performed by the actor.


Harvey MD. Models for Accident Investigation. Workers Health, Safety and Compensation, Occupational Health and Safety Diovision. Alberta, Canada. April, 1985.

Keong TH. Accident analysis techniques. Multilinear Events Sequencing.

MES

Accident sequence begins at to

Stable situation is disturbedBeginning of the act which had to be detected,

adapted, corrected, or otherwise changed for the course of events to have had a different outcome

Accident sequence ends at tn

Last consecutive harmful event connected directly with the accident


MES

Adapted from: Benner L. Accident investigations: Multilinear events sequencing methods. J Safe Res. June 1975. Vol. 7. No. 2.

to = 11:01 tn = 11:02

MES

AdvantagesIncludes temporal relationship of eventsLimits focus to the accident rather than focusing

on managementHas been called the best model available by some

investigators

DisadvantageFocuses almost exclusively on the accident and

ignores management

Harvey MD. Models for Accident Investigation. Workers Health, Safety and Compensation, Occupational Health and Safety Division. Alberta, Canada. April, 1985.

STEP Developed by Hendrick and Benner in 1987 Refinement of the MES technique Each actor’s actions are traced from the start of an

accident to the finish Actor + Action: Who (person or object) must do what to

produce the next event

Events are positioned along a timeline Causal links are represented by arrows connecting

events Includes quality control with sufficient logic testing to

assure consistency and validityLivingston AS, Jackson G, & Priestly K. Root causes analysis: Literature review. Health and Safety Executive. Birchwood, Warrington. 2001.

NASA. QS/Safety and Risk Management Division. Procedures and guidelines for mishap reporting, investigating, and recordkeeping. NPG:8621.1. June 2, 2000.

STEP

Livingston AS, Jackson G, & Priestly K. Root causes analysis: Literature review. Health and Safety Executive. Birchwood, Warrington.

ECFA

Identifies causal factors for each significant event in an accident sequence

Designed as a stand-alone technique but most effective when used with other methods (i.e. MORT, RCA)

No “timeline” but temporal relationships are accounted for

Buys JR, Clark JL, Kingston-Howlett J, and Nelson HK. Events and causal factors analysis. Scientech, Inc. Idaho Falls, ID. August 1995.

ECFAEvaluate events to determine significant events:

The accident would not have occurred if the significant event had not occurred

The event deviated from what was planned or intended

The event had unwanted consequences

Determine the causal factors that allowed each significant event to occur:Who, why, what and how?

Department of Energy Accident Investigation Program. Events and causal factors analysis. January 19, 2001.

ECFAExample of Accident Chronology

Department of Energy Accident Investigation Program. 1/19/01.

Inspection of rudder PCU deleted from annual inspection

Rudder PCU failure mode not identified

Rudder PCU hydraulics contaminated

Rudder hard-over in-flight19:02:47

Crash 19:03:00

1994 September September 8 September 9

ECFAConditions for 1st Event

Event

4

Event

3

Event

2

Rudder hard-over in-flight

Crash

Crew fails to respond to unusual attitude

Crew fails to analyze unusual attitude

Crew fails to recognize rudder problem


ECFAConditions for 2nd Event

Event

4

Event

3Rudder PCU hydraulics contaminated

Event

1Crash

New maintenance personnel do not detect

Hydraulic fluid becomes contaminated

Change in maintenance services contract


ECFACausal Factors for 1st Event

Event

4

Event

3

Event

2

Rudder hard-over in-flight

Crash

Conditions

Need for UA training unrecognized

Potential need for recognizing rudder problems unrecognized


ECFACausal Factors for 2nd Event

Event

4

Event

3Rudder PCU contaminated

Event

1Crash

Conditions

Need to screen service contract provider unrecognized

Potential for hydraulic fluid contamination unrecognized


ECFA Advantages

Temporal relationships of significant events preserved Ideal for multi-faceted problems with long or complex

causal chain Causal factors for each significant event determined Recommendations easily arrived at from causal factors Helps to identify where deviations from acceptable

procedures occurred Disadvantages

Requires a broad perspective of the event to identify unrelated problems

Time consuming Requires training/and or familiarity with the process

US Department of Energy, Office of Nuclear Energy, Office of Safety Policy and Standards. Root cause analysis guidance document. DOE-NE-STD-1004-92. Washington, D.C. February 1992.

RCARoot Causes are causal factors that, if

corrected, would prevent the recurrence of the same or similar accident.Local Root Causes are specific deficiencies

that, if corrected, would prevent the recurrence of the same accident.

Systemic Root Causes are deficiencies in a management system that, if corrected, would prevent the occurrence of a class of similar accidents.


RCA

Root Cause Analysis (RCA) is a structured procedure to identify and evaluate the underlying causes of an accident to prevent a recurrence.

Goal of RCA is not merely to determine the cause of an accident but to prevent it from occurring again.

NASA, Office of Safety and Mission Assurance, Chief Engineers Office. Root cause analysis overview. July 2003.

Rimson IJ. Investigating “causes” and assigning “blame.” The Investigation Process Research Library. August 2003.

Decision Systems, Inc. What is root cause analysis? Longview, TX. 1999.

RCAProcedure

Phase I: Clearly define the undesired outcome.Phase II: Data Collection. Phase III: Assessment.

Identify the problem and significance of the problem

Identify the causes working back to the fundamental cause, which if corrected, would have prevented the accident (root cause)FTAMORTECFA

NASA, Office of Safety and Mission Assurance, Chief Engineers Office. Root cause analysis overview. July 2003.


RCAProcedure

Phase IV: Corrective actions for each identified cause to prevent recurrence.

Phase V: Follow-up by determining if corrective action effectively prevents recurrence.


RCA

http://www.hq.nasa.gov/office/codeq/rca/rootcauseppt.pdfon

Conclusions

Methodologies largely determined by organization

Methods may be selected Not all methods suitable

for each accident Simplest method that

yields the required results

Frei R, Kingston J, Koornneef F, & Schallier P. Investigation Tools in Context. Noordwijk Risk Initiative Foundation.

accident investigation techniques and methodologies chuck dejohn, d.o., m.p.h federal aviation...

Documents

accident investigations

events benner

modern accident investigation

libris slide

investigator benner

methodology biases

risk analysis tree mort

risk tree mort benner