Language Proficiency

Testing in Aviation – 10 years of

challenges and solutions

Neil Bullock,

International Civil Aviation English Association

Introduction

Communication

‘a code that is used in a

very restricted context and

register associated with

distinctive probabilities of

discourse functions ’ Alderson 2009

Intro to LPRs

• Testing – initial & recurrent

• Pilots and ATCOs

• Speaking & listening only

• 5 Holistic Descriptors

• 6 skills

• 6 levels

• Level 4 minimum

• HIGH STAKES TESTING

ICAO Doc9835

Communicative competence

Bagarić, (2007)

A specific purpose language test ... Allows us

to make inference about a test taker’s

capacity to use language in the specific

purpose domain (Douglas, 2000)

1 – standardisation

2 – it’s a language test!

3 – ambiguity

4 – face validity

5 – testing skills

6 – test driven system

7 – fairness

Key challenges

A group of pilots in Korea

seriously doubted whether

the proficiency test they

were obliged to take

corresponded to the real

tasks and demands of their

job. Kim (2013).

English proficiency …

is only one of the

many factors that

influence performance

in a professional

setting (Kim, 2018)

… miscommunication

between pilots and air traffic

personnel could not be attributed

solely to

limited English proficiency on

the part of non-native speakers

of English (Kim & Elder 2009)

Solution in Testing 1 : TD in CH

The challenges - LP exams for pilots

• Disconnect – TSP / TTs = zero face

validity.

• Poorly produced non-contextual material.

• Little test security.

• No sample material.

• Lack of testing expertise.

• Limited training.

Solution in Testing 1 : TD in CH

The solutions: 2010 >

• Test development team

• Weir / Fulcher /Alderson

• TD plan & Spec

• SMEs – real world + Disc. Analysis

• Training, research and studies.

• Statistics > Analysis > Validity & Reliability

• RTF > Trialling > Samples

• Feedback > Engaged with stakeholders

• 3-year review

Voice only interaction (role play) = 0.84

Positive co-efficients From TT feedback

Face-to-Face Interview = 0.90

Listening Test = 0.91

Exam Organisation = 0.96

After the Exam = 0.94

Overall = 0.91

• 280/551 TTs = 51% response

Standard Setting towards validity 2017-2018

Construct Context Cognitive Content Face Consequential

0.80 0.80 0.81 0.81 0.83 0.81

VALIDITY

Co

nst

ruct

Co

nte

xt

Co

gnit

ive

Co

nte

nt

Face

Co

nse

qu

enti

al

Scores out of 5 (5 = fully agree / 1= Fully disagree)

1 I found the sample test material easy to download ✔ ✔

2 I found the sample test material useful ✔ ✔ ✔ ✔ ✔ ✔

Speaking Ability Exam – VOI Simulated Flight

3 The role-play reflects real operational situations ✔ ✔ ✔ ✔ ✔ ✔

4 The material content for the role-play is appropriate for

the test ✔ ✔ ✔ ✔ ✔ ✔

5 The tasks focus on non-routine & unexpected

operational events for pilots. ✔ ✔ ✔ ✔ ✔ ✔

6 Preparation time for the role-play is appropriate ✔

7 The duration of the role-play is appropriate ✔ ✔

78 TTs (59%) 31 questions

• 30+ years

• 600 members in 80 countries

• Board – Pres + 4 VPs

• Support with testing and training

• Involvement with ICAO/EASA

• Presentations & Papers

• Social media & Linked in forum

Solutions 2 – ICAEA

Solutions 2 – ICAEA Projects

• Test Design Guidelines project

• https://drive.google.com/open?id=

1VJvzXTAxkt-zxJwxrJOJNQl-

aUyWwu-W

• Help & guidance:

o LPR test design

o Select and audit tests

o Design & improve tests

o Confidence and face validity

o Standardisation

ICAEA Workshops

Roll-out phase:

1. 2019 and 2020 (2.5 days / 50 max):

Regional participation

• Europe (Luxembourg)

• Asia (Thailand)

• Americas (Argentina)

• Middle East (TBC)

• Africa (TBC)

2. TDGs (published online): 2019

3. ICAEA support – ongoing 2019/20

The Future of the LPRs & testing…

• Assessment literacy

• Domain communication

• Interactive competences

• Socio-cultural competences

• L1 awareness

• Research, papers & books chapters

• Using skills and expertise

• Engaging with all stakeholders

• Building relationships EASA/ICAO/TF

Thank you & bon voyage

• neilbullock@icaea.aero

References

• Alderson, C. J. (2009). Air safety, language assessment policy, and policy implementation: The case

of aviation English. Annual Review of Applied Linguistics, 29, 168–187.

• Bullock, N. (2015). Wider considerations in teaching speaking of English in the context of

aeronautical communications, IATEFL ESPSIG Journal, 45, 4-11.

• Bullock, N. (2015). Defining meaningful material for the teaching of English for

• aeronautical communications, in A. Borowska & A. Enright, A. (Eds.) Changing perspectives on

Aviation English training. Warsaw : Uniwersytet Warszawski.

• Douglas, D. (2000). Assessing language for specific purposes, Cambridge: CUP.

• International Civil Aviation Organisation. (2010). Doc 9835 Manual on the implementation of ICAO

language proficiency requirements. Montreal: ICAO.

• International Civil Aviation Organisation. (2007). Doc 9432 Manual of radiotelephony. Montreal: ICAO.

• Kim, H & Elder, C. (2015). Interrogating the construct of aviation english: Feedback from test takers in

Korea. Language Testing, 32 (2), 129-150.

• Kim, H. (2013). Exploring the construct of radiotelephony communication: A critique of the ICAO

English testing policy from the perspective of Korean aviation experts. Papers in Language Testing

and Assessment, 2 (2), 103–110.

• Moder, C. & Halleck, G. (2009). Planes, politics and oral proficiency: Testing international air traffic

controllers. Australian Review of Applied Linguistics, 32(3).

• Paltridge, B. & Starfield, S. (2013). The Handbook of English for Specific Purposes, Chichester: John

Wiley and Sons Inc.

• Weir, C.J. (2005) Language Testing and Validation – An Evidence Based Approach. London:

Palgrave Macmillan.