An Analysis of Operational Errors and the interaction with TCAS

7th ATM R&D Seminar – Barcelona (Spain) – July 2-5, 2007

July 2-5 , 2007

An analysis of Operational Errors:

Interaction with TCAS

Kevin M. Corker, PhDProfessor, San Jose State University & Jose L Garcia-ChicoATC Research AnalystTitan Industries

July 2-5 , 2007Slide 2

Agenda

1

4

3

2

Problem Statement - Motivation

Methods

Results

Conclusions

TCAS system and operators’ behaviour

5

July 2-5 , 2007Slide 3

Problem Statement

• Operational Error (OE) rate has been increasing through 2003 and reaching a plateau in the US airspace. The absolute number of OEs is still increasing.

• Specific Interest: recent accidents/incidents involving TCAS

• For Example

• Yaizu (2001), TCAS involved in both a/c. No fatalities

• Uberlingen (2002), TCAS involved in both a/c. 71 fatalities

FAA (2006, April). Administrator’s Fact Book. Washington, DC: Department of Transportation.

OE rate per 100,000 facility activities

0.52 0.52 0.530.51

0.56

0.6

0.69

0.74

0.66

0.78 0.78 0.77

0.4

0.6

0.8

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

12111216 1506

July 2-5 , 2007Slide 4

Motivation & Ongoing Study

• Classify operational errors and contextual factors in ATC in search of trends and consistency in the classification.

• Assumption: classification of errors provides understanding of work performance and organizational/operational context.

• Focus on presence of TCAS RA in evolution of the operational error

• TCAS is an effective safety system, with caveats…

� It might disrupt the controller’s SA (Brooker, 2004; Wickens et al, 1998)

� Amplified by the fact that changes FL (vertical resolution) are only provided as number in data block)

� It might create inconsistent pilot and controller responses (Rome et al., 2006, Wickens et al, 1998)

� Intention: Understand procedural and informational context of OEs co-occurring with TCAS RAs.

July 2-5 , 2007Slide 5

TCAS – expected behavior

• For TCAS to work as designed, immediate and accurate crew response to TCAS advisories (action within 5 sec.) is essential.

• Regulation of TCAS: operational procedures and practices (FAA AC 120-55B)

Pilots:

• Should follow TCAS RA, unless doing so would jeopardize the safe of operation. (Required response within 5 sec of RA display)

• During an RA, do not maneuver contrary to the RA based solely upon ATC instructions.

• S/he has to report any deviation from ATC clearance, as soon as practicable after responding to the RA, and resume previous clearance after “clear of conflict”

Controllers:

• Will not knowingly issue instructions that are contrary to RA guidance when they are aware that a TCAS maneuver is in progress.

July 2-5 , 2007Slide 6

TCAS events timeline (“desired”)

Adapted from: Brooker, P. (2004). Thinking about downlink of resolution advisories from airborne collision avoidance systems. Human Factors and Aerospace Safety 4 (1), 49-65.

Timeline ATC SA Impaired ATC aware deviation

Pilot notifies ATC of

deviation

Pilot notifies

Return to clearance

RA

Pilot follows RA &

deviates from clearance

Window to receive ATC

clearance in opposition to RA

without controller aware of RA

Controller provides traffic info,

If workload permits

Controller is not responsible for separation

Clear of conflict

Window to receive ATC

clearance in opposition to RA

with controller aware of RA

July 2-5 , 2007Slide 7

Other TCAS research: Operator behavior during TCAS

• TCAS in simulation settings (Rome et al., 2006)• Variability and deficiencies in pilot communications• TCAS RA maneuvers increased stress • Controller cleared vertical deviations during RA maneuvers (4

out 32).

• Research on RA downlink (Brooker, 2004; Eurocontrol, 2003b, 2004)

• Controllers found it beneficial: – Improve SA and – avoid contradictory ATC clearances.

• Problematic issues: – overload of information, – pilot compliance, – change in responsibilities, – procedures, – liability.

July 2-5 , 2007Slide 8

Methods

• Exploratory Study: mapping relationships in the data.� Analysis of errors based on preliminary and final Air Traffic

Controller Reports

• Excluded: Surface and Oceanic Errors

• Two studies/datasets:� Taxonomic Study: classification of OE initial incident

reports (Jan-Jun 04 period: 480 OE reports) • Classification of OEs based on FAA classification schema.• Relevance of coordination, training, proximity, time on position.

� Focused Study: OEs with presence of TCAS RAs. Final reports (Jan-Jun 2004 & 2005: 62 reports)• Use of same classification.• Characterization of the TCAS RA events. Human response.

July 2-5 , 2007Slide 9

Results Study 1: Taxonomic Study

810480TOTAL

560 (69.1%)318 (66.04%)ARTCC

250 (30.9%)162 (33.96%)Terminal Radar

Operational Error Classification

Operational Errors Reports

OE Classification

2832

9

22 25 24

7

22

10 813

10 82

12

41 1

41 1 1 0

5

84

44

66

4843 41

49

3236

2317 16

813

37 9

51 3 2 1 3

6

0

10

20

30

40

50

60

70

80

90

Fail C

onve

rging

Contro

l coo

rd

Desce

nd tr

houg

h

Overlo

oked

Trf

Vecto

r ina

dequ

Hear/R

eadb

ack

Altitud

e In

adeq

u

Fail A

lt Clim

b/Des

cend

Climb

thro

ugh

Fail O

verta

king-

Trf

Instr

uc n

o-int

ende

d

tem

p er

ror-i

ssue

Misa

ppl P

roce

d

data

block

-mise

nter

Airspa

ce

Trans

pose

a/c

FPS-mise

nter

Speed

inad

equ

Wro

ng a

/ca/

c ove

rlap

LOA m

is

Cleare

d blw

min

Misr

ead

info

othe

rs/w

hat

TRACONARTCC

July 2-5 , 2007Slide 10

Proportion of OE types (based on the total OE number)

13.8%

9.4% 9.3%8.6% 8.4% 8.0%

6.9% 6.7%5.7%

3.8%3.2% 3.7%

18.3%

4.0%

10.9%

9.4%

7.9%

3.5%

8.4%

11.9%

8.9%

1.0%

5.0%

2.5%

0%

10%

20%

Fai

l Con

verg

ing

Con

trol c

oord

inat

ion

Des

cend

thro

ugh

Ove

rlook

ed T

raffi

c

Vec

tor I

nade

quat

e

Hea

rbac

k / R

eadb

ack

Alti

tude

Inad

equa

te

Fai

l Ide

ntifi

catio

n

Alti

tude

Clim

bing

/Des

cend

ing

Clim

b th

roug

h

Fai

l Ove

rtaki

ng T

raffi

c

Tem

pora

l error

-issu

e

Inst

ruct

ion

not

Inte

nded

Full set of reports TCAS RA reports

OEs co-occurring with TCAS

7826TOTAL

44 (56.4%)18 (69.2%)ARTCC

34 (43.6%)8 (30.8%)Terminal Radar

Jan-Jun 05Jan-Jun 04

July 2-5 , 2007Slide 11

ATC Commands IN TCAS Situations

ATC Commands In RA OE

0

5

10

15

20

25

30

35

40

45

Before RA After RA None Undetermined

% o

f 10

4

Series1

July 2-5 , 2007Slide 12

ATC Commands Dependence on Information Integrity

Information Integrity

0

5

10

15

20

25

30

35

40

Horizontal:BeforeRA

Vertical: BeforeRA

Horizontal:After RA

Vertical: AfterRA

None

% o

f 59

Rep

ort

s

Complete Info

Incomplete Info

July 2-5 , 2007Slide 13

ATC Vertical Commands after RA and Flight Deck Report

Vertical Commands After RA

0

10

20

30

40

50

60

Vertical Correct Vertical In Opposition

% C

orr

ect

ou

t o

f 20

Series1

July 2-5 , 2007Slide 14

Deviations from “expected” behavior

Clearances issued by controller upon triggered TCAS RA

0

5

10

15

20

25

Before After None Before After None Before After None

% o

f in

cid

ents

TrafficHeadingAltitude

COMPLETE INCOMPLETE NO REPORT

9.78.1

12.9

4.8 17.7

14.5 12.9

8.1

1.63.2

6.5

0 opposite to RA 4.8 opposite to RA

Incomplete = missing any pilot’s message, missing callsign, TCAS direction or excessive delay

Before and after refers to the action of controller in relation to the TCAS RA event.

Traffic, heading, or altitude mean ATCO gave traffic info, or change heading, or altitude

July 2-5 , 2007Slide 15

Highlights on the chain of events during TCAS RA encounters in OE reports

• Controllers issued clearances after TCAS RA in the vertical plane in 13 situations (21 %).

• Controllers received incomplete information in 26 situations (43.5%) and no information in 3 (5%). Opportunities for wrong decisions.

• Controllers issued vertical clearances after TCAS RA and incomplete pilot’s reports in 12 situations (19.4 %).

• opposite altitude clearance in 3 reports (4.8%)– Pilot reports were all late after TCAS RA and controller clearance

• Data suggests that it is more likely to receive an opposite clearance if the controller receive incomplete pilot information.

July 2-5 , 2007Slide 16

Proposed Actions

• Increase training recreating TCAS RA situations� Under stress situation, abnormal events trigger more familiar responses (i.e.,

issue vertical clearance)

• Revisit downlinking RAs� Future research needed

� Not obvious solution, with important implications• Draw too much controller attention• TCAS RA is not the most relevant information, but the pilot deviation from

clearance• Controller’s responsibility and liability implications

July 2-5 , 2007Slide 17

Conclusions

• Value of systematic characterization of errors� OE classification would allow prioritization of actions.� Failure to notice converging aircraft, control coordination,

hearback/readback, and overlook traffic are the most frequent

• Error reports concurrent with TCAS RA:� OEs with similar patterns to full dataset� Not consistent pilot-controller behavior (deficient information/actions)� Incomplete/late information increases chances of vertical clearances

incompatibles with RA direction

July 2-5 , 2007Slide 18

Acknowledgement

• Special thanks for comments on this paper and insightful ideas during the study to Dr. Kim Cardosi (Volpe Laboratories) & Ms. La Gretta Bowser (FAA)

• Thanks to Mr. Bill Davis (OSTP) for his sponsorship and comments

July 2-5 , 2007Slide 19

Questions

July 2-5 , 2007Slide 20

Back Up slides

July 2-5 , 2007Slide 21

Proximity Rating

• Proportion of higher-proximity events in terminal areas.• Errors with low frequencies have higher proximity (reduced

cross check)

(Chi-square X2 (2,N=460)=226, p<0.001)

16 (10.5%)5 (1.5 %)No rated

21 (13.7 %) 285 (86.9%)Proximity Rating C

65 (42.5 %)31 (9.5 %) Proximity Rating B

51 (33.3%)7 (2.1 %)Proximity Rating A

TRACON ARTCC

July 2-5 , 2007Slide 22

Error Severity and Frequency by Time on Shift

• No statistical significance in the distribution of frequencies (60 min.)

• Not been able to claim that errors are more likely after break relief or transition into position.

• No evidence that errors were more severe in the first 30 minutes after taking over control (X2 (10,N=373)=7.27, p=0.700)

38

33

37

3028

3735

32

35

32

24

27

18

1113

108

7

4 4

1

4 5

0

5

10

15

20

25

30

35

40O

E F

req

uen

cy

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 110 120 >120

Minutes on Position

(Chi-square X2 (11,N=388)=6.575, p=0.832)

July 2-5 , 2007Slide 23

TCAS RA and the proximity of aircraft

• Higher proximity when TCAS RA is triggered (only in centers we could proof statistically)

• Smaller than expected.� Consequence of time logic implemented by TCAS, and/or

� Proof of global efficiency of TCAS (“safe the day”)

Proximity rating of OE

13.5% 10.6%4.8% 1.5%

18.3%13.5%

14.4%

4.4%

7.7%

6.5%

40.4%59.4%

1.0%

3.1%

0.0%

2.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

TRACON (RA) TRACON ARTCC (RA) ARTCC

Unk

C

B

A

Centers (Chi-square=32.037, p<0.001)

TRACON (Chi-square=0.254, p=0.88)