aviation safety reporting system (asrs): what … hf_sms seminar...july 23, 2013 aviation safety...

July 23, 2013

Aviation Safety Reporting System (ASRS):

What Information Tells Us

Linda Connell, NASA ASRS Director

Human Systems Integration Division

NASA Ames Research Center

Aviation Human Factors and SMS

Seminar 2013

Frontiers of Flight Museum, Dallas TX

Moffett Field - Hangar One

1932

Moffett Field - Hangar One

Aviation Safety Reporting System

ASRS Contributions to a System Safety Process

ASRS disseminates vital information to

the aviation community

This function is specifically designed to

help the aviation community recognize

accident precursors and take preventive

action

ASRS promotes the ability of safety

professionals to conduct more

prospective safety analysis

By evaluating the experience of other

reporters in the system, safety

professionals can learn from other reporters in similar situations without having to have to experience the undesired outcome for themselves.

Source: http://www.faasafety.gov/gslac/ALC/libview_normal.aspx?id=6877

Aviation Safety Reporting System Aviation Safety Reporting System

ASAP Review Committees could include an ASRS DataBase OnLine (DBOL) search of other events from a larger sample of similar events with ASRS Analysts providing preliminary analysis coding on numerous variables (contributing factors, human factors, etc).

As SMS procedures are implemented, an ASRS search could assist in efforts to discover information from additional data sources.

Topics for safety training can be enhanced with use of actual narratives describing events.

Newsletter and publication content can be expanded using illustrations of ASRS events.

Opportunities to Use ASRS Safety Data

Aviation Safety Reporting System Aviation Safety Reporting System 6

Since ASRS is a national repository of industry safety data, an ASAP Event

Review Committee (ERC) can rely on ASRS data to provide a systemic view

into the larger picture of emerging issues and hazards

The ERC team can use ASRS data to review the experiences of other

operators to help identify issues they have not yet encountered, or recognize

weaknesses in their own safety programs that may not have yet materialized

This may provide them the ability to detect potential hazards and consider

safety solutions which are discovered in reports submitted by similar operators

ASRS data can provide valuable insight into investigating factors that

contributed to a past event or provide insight into researching the potential of a

future issue

An open minded, forward looking approach to exploring potential hazards is

complementary to the carrier’s SMS program

What can ASAP ERCs gain from ASRS?

Aviation Safety Reporting System Aviation Safety Reporting System en Safety Reporting System

Database Searches

Direct access to an automated data search tool will provide de-identified reports from the ASRS Database. This is available online through the ASRS (http://asrs.arc.nasa.gov/search/database.html).

Information in the ASRS Database is publicly available. The ASRS can provide assistance upon request through Search Requests. ASRS will search its database, download relevant reports, and send to the person/organization requesting information.

For your convenience, selected relevant reports on several pre-selected safety topics are available at ASRS Database Report Sets. (http://asrs.arc.nasa.gov/search/reportsets.html).

The ASRS Database is also available through the FAA Aviation Safety Information Analysis and Sharing (ASIAS) website at http://www.asias.faa.gov/. This is updated monthly from ASRS.

Voluntary data are qualitative data

Data Typology

Qualitative

Describes the “why” and “how”

of events

Diverse responses through

narratives and storytelling

Captures themes, observations,

environmental context that

generates informed assertions

Holistic and contextual

Focus on in-depth

understanding of human

behavior and the reasons that

govern such behavior

Quantitative

Describes the “what” or “how

often” of events

Narrow categorizations for

unbiased measurements

Hypotheses driven focus on a

specific question and collects

sampling of numerical data

Reductionist and objective

Serves statistical, applied

inferential mathematics, or

computational techniques

Qualitative vs Quantitative Data

ASRS reports are subjective recollections of events experienced or witnessed by professionals operating in their specific domain

As a data source, these reports are characterized as qualitative and voluntary, which provide insights into the “why” of events from people operating within the system

Qualitative and voluntary data compliments quantitative sources that describing the “what” and “how often” an issue occurs in the system

Qualitative and voluntary systems provide a means to capture candid disclosures of the human factor contributions to safety events and can offer validation as to whether a mitigation has eliminated anomalies in the system

Statistical Nature of Voluntary Subjective Data

This data is not a random sample from the population of all events

Qualitative analyses might be appropriate, if properly constructed

Hypothesis generation possible for future research

Traditional quantitative statistics not appropriate (significance results,

trending, probability, frequency, correlation, etc.

Misuse of qualitative data in statistical analysis can

produce subtle, but serious errors in description and

interpretation –

Subtle in the sense that even experienced professionals can make

such errors, and

Serious in the sense that they can lead to decision errors.

S

Standards For Using Qualitative Data

Voluntary Data and Statistical Bias

Bias in Voluntary Sampling • In voluntary sampling, bias refers to the tendency of a sample statistic to

systematically over- or under- estimate a population parameter.

Non-response bias • Non-response bias results when respondents differ in meaningful ways from non-

respondents. Non-response is often problem with voluntary input, where the

response rate can be very low.

Voluntary Response Bias

• Voluntary response bias occurs when sample members are self-selected volunteers,

as in voluntary sampling. An example would be call-in radio shows that solicit

audience participation on controversial topics (affirmative action, gun control, etc.).

The resulting sample tends to over-represent individuals who have strong opinions.

Source: http://stattrek.com/statistics/dictionary.aspx

Implicit Biases of Voluntary Reporting

Example: Near Miss

Nonresponse bias

Voluntary

Response Bias

ASRS Past, Present and Future



FAA and NASA Partnership

MOA signed by Administrators for FAA and NASA

. . . To provide information to the FAA and the aviation community to assist them in reaching the goal of identifying and eliminating unsafe conditions to prevent accidents.


ASRS Gov’t/Industry Stakeholders

FAA provides reimbursable

funding to NASA for ASRS

Support & Management

NASA provides funding for

Director to provide overall

management

• Assures independence from

perceptions of potential conflicts of

interest

• Reinforces role of trust in success

The Aviation Community provides

support through advocacy for

reporting, feedback, and

communications


System Recognition for Effectiveness

US News and World Report 2006 praises system FAA credited for a positive, proactive approach to safety

Proof that government agencies don’t have to act

like big-footed oafs, the FAA (and NASA) has a

system that allows pilots and air traffic controllers

to report problems anonymously, It’s a “Let’s

learn, let’s fix it” sort of culture. The goal is to

find systemic problems and solve them rather

than assign blame. Most errors and mistakes are

caught early, and accidents are rare.

Constellation of Aviation Safety Efforts

ASAP

FOQA

ASIAS

VDRP

LOSA

ASRS

CAST

HFAC

ASRS Overview and Status


Aviation Safety Reporting System 19

ASRS Purpose and Mission

Identify deficiencies and

discrepancies in the National

Airspace System

Provide data for planning and

improvements to the future

National Airspace System

Precursors


ASRS

FAA & NTSB

Normal Operations

Incidents

Injury

Accidents

Fatal Accidents

System-Wide Event Occurrences

ASRS is complementary to other systems of reporting

and focuses on precursors to the most severe events


Report Processing Flow

ASAP and ATSAP Reports Enter

(Electronic and Paper)


Incident Reporting Model

ASRS is a closed

loop process that

supports system

safety and Human

Factor insights

Government /

Industry are

provides

information that

may result in

corrective actions


U.S. Aviation Statistics *

FAA Aviation Personnel * • Pilots 618,707 • Air Traffic Controllers 14,305 • Dispatchers 21,664 • Mechanics 314,931 • Flight Attendants 170,155

*July 2012 FAA Certification Database

** 2011 Bureau of Labor Statistics

*** RITA Statistics

Active Aviation Labor Force

**

• Pilots - Commercial/ATP 99,980

• Aircraft Mechanics 35,070

• Flight Attendants 87,190

Potential Aviation Reporters

TOTAL (Est.) 1,139,795

Flight Volume ***

62,000 Flights/Day (Air Carrier, Cargo, Military)

27,178 Flights/Day (General Aviation)

Monthly Intake

January 1981 – December 2012

ASRS Report Volume Profile

37 years of confidential

safety reporting

Over 1,000,000 reports

received

Over 5,550 alert

messages issued

Over 6,000 reports per

month, or 300 per

working day

Total report intake for

2012 was 71,540

Current rate estimate

for 2013 is over 75,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

'81'82'83'84'85'86'87'88'89'90'91'92'93'94'95'96'97'98'99'00'01'02'03'04'05'06'07'08'09'10'11'12

Smoothed Intake


ASRS Reporting Community



• Electronic (secure electronic transfer protocols)

• Direct from Airline Program

• WBAT System or other software

• Direct from ATSAP

• Paper Forms

• Continue to be used by some ASAP airlines

Electronic Submission (ERS) from website

Form on website

• Fill out on computer, print, mail

• Print, fill out by hand, mail

Paper Forms

• Remains a continuing source of reporting

• Diminishing volume of paper

Direct-to-ASRS ASAP/ATSAP-to-ASRS

ASRS Report Options


0%

20%

40%

60%

80%

100%

2003

34,043

2004

38,116

2005

40,657

2006

39,694

2007

45,603

2008

50,405

2009

48,986

2010

58,683

2011

61,018

2012

71,540

Other Maintenance Cabin Crew Gen Av Air Carrier Controller

Incident Reporter Distribution

January 2003 – December 2012

Percentage of Total Intake

Air Carrier Reporting 61%

General Aviation 18%

Controller Air Carrier General Aviation Cabin Crew Maintenance Other

Report Groups Intake 2001- 2012

Decreases evident following September 11, 2001 are showing return to pre-9/11 levels

20% of all reports are matched to unique events


Incidence of ASRS Multiple Reports

20%

100%

Provides information from more than one person’s

perspective on a single event

Co-Pilot

Pilot

GND 1

Other 1

Maintenance

2

Aviation Example:

Flight Attendant Report Matching

The ASRS Database contains a total 3,533 Flight Attendant reports. 820

of those reports are matched to at least one other report. These 820 reports

correspond to a total of 432 unique incidents.

ATC 1

Pilot 181 Flight Attendant

820 Flight

Attendant

605

432 Unique Events

(1,006 Total Reporters)

ASRS Report Intake – ASAP / ATSAP

Overall ASAP Intake

• 176 Total Programs

• 75 Air Carriers

Reporting Groups

• 73 Pilot

• 43 Mechanic

• 38 Dispatch

• 18 Flight Attendant

• 4 Ground Crew

Secure Electronic Data connection protocols between airline and ASRS

• 174 Programs

• 74 Airlines

ASRS Electronic Transmission

Methodology compatible with

numerous software platforms

More airline programs are

being added continuously

Report Source - ASAP/ATSAP and Direct

0

1000

2000

3000

4000

5000

6000

7000

Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12

ASAP/ATSAP Direct


ASRS Products

These products and services fulfill the program’s

mission to disseminate safety data


Alert Message Coordination

Reminder: Definition of Alert Bulletins (AB)

• Brief, single-topic messages issued when reported safety

problem is well-documented and involves serious safety

concern. ABs must meet several criteria:

• Report(s) which form subject of AB must be describing

significant safety concern

• Alert Bulletins always have one “major” addressee as recipient,

and frequently are sent to multiple “information” addressees at

time of initial issuance.

• Major addressee is authority or organization in best position

to explore alleged safety concern and to correct problem.

• Informational addressees (cc) are other “interested” organizations and individuals in positions to convey

essential information of AB to their memberships,

constituencies, or managers.

Reminder: For Your Information Notices

• Brief, single-topic messages issued when report information is

fragmentary, or when the safety problem is judged to be less

severe, but important.

• FYI notices use same distribution conventions as AB for

addressees.

How Potential Alerts Are Processed

Alert Coordinator

Screening of

Red Tag Items

Database Search

For Similar

Incidents

Determination

of AB/FYI or

TELECON

Expert Analyst

Analysis &

Call Back

Alert Item

Tracking Forms

Attached

Distribution of

Alert Message

Collect and

Evaluate

Feedback

Alerts are identified at any point from Screening to

Callback

ASRS Alert Messages

Airport

28%

Manufacturer

29%

FAA

43%

ASRS Alert Message Recipients - 2012

FAA Voluntary Safety Programs Branch (AFS-230)

Receives every Safety Alert

n = 217

ASRS Alerting Messages January 1999 – December 2012

153

250

282

212 214

304

208192

342

275

236

265

201217

0

50

100

150

200

250

300

350

400

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

CY Year

n = 3,351

313

56

72

121

137

165

227

250

341

568

1,101

0 200 400 600 800 1000 1200

Other

Aircraft Avionics

Navigation

Aircraft Powerplants

Hazards to Flight

ATC Equipment

Airport Lighting and Approach Aids

ATC Procedures

ATC Operations

Airport Facility Status and Maintenance

Aircraft Systems

Alerting Subjects January 1999 – December 2012

n = 3,351

Alert Responses

Response Percentage

Action taken as a result of the AB/FYI 27%

Action initiated before AB/FYI received 11%

Action initiated in response to AB/FYI but not

completed 11%

Issue raised by AB/FYI under investigation 6%

Addressee agrees with AB/FYI but sees no problem 6%

Addressee in factual agreement but is unable to

resolve 3%

Addressee disputes factual accuracy of AB/FYI 19%

Information in AB/FYI insufficient for action 10%

For information only, no response expected 4%

Action not within addressee's jurisdiction 3%

64%


1999 – 2012

ASRS Alert Messages and Feedback


ASRS Alert Messages and Feedback

“The most troubling aspect of LED lighting concerns

is that they are often being discovered through

voluntary disclosure programs such as the Aviation

Safety Reporting System (ASRS). This strongly

suggests that not all of the unintended consequences

of LED lighting implementation have been discovered.

This conclusion alone should encourage the FAA to

take a more measured approach to LED lighting

implementation.”


A Detailed Example of a Recent Alert

This Alert highlighted a critical flight safety anomaly

This anomaly had gone unnoticed during design,

certification, manufacture, oversight, and operations

in service

No SDR was issued since no event had yet occurred

An AD note was issued for a sister fleets but not for

this, newer fleet type that had the same design

configuration

ASRS was able to detect this anomaly though a single

report with no operational incident (weak signal)


Did We Get Our Wires Crossed?

The configuration of the fire extinguishing agent

plumbing system can pose a potential challenge to

maintenance personnel

Several design features appear to have been used to

mitigate a possible system misconfiguration

However, this report shows it is still possible to

install the components in the wrong position

The reporter indicated that “in the worst case, if both

bottles were affected, neither engine would have fire

protection, with extreme risk for loss of life.”


Embraer E170 Fire Bottle Installation Report 1050457

Normal Configuration

This graphic is for illustrative purposes only and not to be used for any other purpose.

Squib

1

A1 B1


Engine 1 Fire

Discharge Shot A


Engine 1 Fire

Extinguishing System

Abnormal Configuration

Engine 1 & 2 Fire

Extinguishing System

1

A1 B1

B2 B1

This graphic is for illustrative purposes only and not to be used for any other purpose



Currently, there are no applicable Airworthiness

Directive or Service Bulletin for E170 series aircraft

AD 2004-23-19 is effective for E135 and E145 series

aircraft to correct the design anomaly

• General visual inspection to detect incorrect wiring

• Modify the electrical harnesses by adjusting the length

• Install identification sleeves on the harness and

connectors and matching color-coded stickers on the fire

bottles


• Structured Callback Studies

(funded by FAA)

• Weather Data Link

• Wake Vortex Encounter

• General Aviation Loss of Control • Accident/Incident Comparison

• RNAV – STARS and SIDS (TBD)


Special Studies and Projects


Meteorological and Aeronautical Information Services Data Link Services and Applications Study - ASRS is gathering reports of incidents that occurred while pilots were utilizing weather or AIS information in the cockpit (textual and/or graphical) obtained via data link (including ACARS) or other sources on the ground or in the air.

Wake Vortex Encounter Study - ASRS began this study in 2007 and will continue through 2013. Wake Vortex Encounter Study includes all airspace within the United States, en route and terminal. In quarterly reports, the ASRS documents event dynamics and contributing factors underlying unique wake vortex encounter incidents.

ASRS Conducts Studies on Special Topics


ASRS Web Site

Launch Oct. 2007

• Over 9 million

sessions in 2012

File ASRS Report

• Electronic

• Print and Mail

Database Online

ASRS Publications

Program

Information

Immunity Policies

http://asrs.arc.nasa.gov


ASRS Database Online (DBOL)

DBOL launched August 23,

2006

• Over 118,000 total online queries completed to date

• Over 19,600 queries completed in 2012

Fixed field and text search capability

Data formats (export)

• MS Word, Excel, CSV HTML

Experts version (DBOL II) being proposed


July 23, 2013

ASRS Model Applied to

Aviation & Other Industries


International Confidential

Aviation Safety Systems (ICASS)

South Korea

KAIRS (2000) United States

ASRS (1976)

United Kingdom

CHIRP (1982)

Canada

CASRP (1985)

SECURITAS (1995)

Australia

CAIRS (1988)

REPCON (2006)

Russia

VASRP (1992)

Brazil

RCSV (1997)

Japan

ASI-NET (1999)

Taiwan

TACARE (2000)

Singapore

SINCAIR (2004)

France

REC (1999) China

SCASS (2004)

New Zealand

ICARUS

South Africa

SASCO

Germany

EUCARE

Spain

SNS (2007)


ASRS - Model for Confidential Reporting

There is a growing interest in utilizing the

ASRS reporting model for application to other

disciplines such as medicine, fire fighting,

railroad, maritime, security, and others.


ASRS Model Applied to Other Industries

• Confidential Close Call Reporting System (C3RS) • Railroad Safety Reporting System was modeled after ASRS

• Under development at NASA ASRS through collaboration with

Federal Rail Administration and Volpe National Transportation

System Center

• Fire Fighters Near Miss Reporting System • Launched August, 2005 was modeled after ASRS

• Development Task Force included FAA and NASA ASRS

• Bureau of Safety and Environmental Enforcement

(BSEE) • New Interagency Agreement for offshore oil and gas industry safety

reporting system


NASA ASRS Model Applied

to Railroad Confidential

Close Call Reporting System

(C3RS)


BSEE Safety Reporting System


Unique Aspects of ASRS Confidential

Reporting Model

System-Wide Perspective - capability to identify hazards identified by aviation personnel and match reports from all segments of aviation community

• ASRS was catalyst for recent FAA focus on Teterboro Departures

System-Wide Alerting - both national and international capability to provide ASRS Alert Messages to industry and government

Data Processing through Aviation Expert Analysts

• ASRS Office staff include Aviation Expert Analysts with a combined total of 380 years of experience in aviation (air carrier pilots, corporate pilots, general aviation pilots, air traffic control, and maintenance)

• Experts read and review 100% of reports and reliably code information to databases

Comprehensive and Time Tested Coding Taxonomy

• Fixed Field Codes combined with Narrative Text yields qualitative data for further secondary analysis techniques (Perilog, special studies, focused analytic techniques, etc)


Strong Immunity and Legal Provisions

• Federal Law specifically addressing ASRS (14 CFR 91.25)

• FAA Advisory Circular 00-46E

• ASRS Addressed by Congress in 1980’s

Information Sharing - both nationally and internationally with industry and government

• Database Search Requests, Database Publically Available, Topical Studies, Structured Telephone Callback Studies, Collaborations with Industry and Gov’t (FAA, NTSB, NASA, TSA, etc.)

• Largest source of airline ASAP data collected in central location

National and International Reputation

• ASRS Recognized Model for Proactive Contribution to Safety Process

• ASRS Model Being Utilized by Other Domains for Safety Improvements

Unique Aspects of ASRS Confidential

Reporting Model


Contact Information:

Linda Connell, NASA ASRS Director

[email protected]

(408) 541-2827 ASRS Office

(650) 604-0795 NASA Office

ASRS Website:


mailto:[email protected]

aviation safety reporting system (asrs): what … hf_sms seminar...july 23, 2013 aviation safety...

Documents