university of michigan english language institute testing and certification division behavioural...

University of MichiganEnglish Language Institute

Testing and Certification Divisionwww.lsa.umich.edu/eli

Behavioural scales of language proficiency: insights from the use of the Common European

Framework of Reference

Spiros Papageorgiou

14th International GALA conference, Thessaloniki, 14-16 December 2007



Outline • Background

• Aims

• Data collection

• Data analysis

• Results

• Implications



Background • Advent of the CEFR: increased interest in

behavioural scales of language proficiency

• Using the CEFR scales: Problems

Designing test specifications (Alderson et al., 2006) Measuring progression in grammar (Keddle, 2004) Describing the construct of vocabulary (Huhta &

Figueras, 2004) Designing proficiency scales (Generalitat de Catalunya,

2006)



Background (2)• Using the CEFR scales: Criticism

Equivalence of tests constructed for different purposes (Fulcher, 2004b;Weir, 2005)

Danger of viewing a test as non valid because of not claiming relevance to the CEFR (Fulcher, 2004a)

Progression in language proficiency not based on SLA research but on judgements by teachers (cf. North 2000; North & Schneider 1998)



Aims of the study• Investigation of three research questions:

Can users of the CEFR rank-order the scaled descriptors in the way the appear in the 2001 volume?

If differences in scaling exist between the users of the CEFR and the 2001 volume, why does this happen?

Can training contribute to more successful scaling?



Data collection• 12 users of the scales acting as judges in relating

two language examinations to the CEFR

• Data collected during Familiarisation sessions described in the Manual for relating examinations to the CEFR

• Part of a doctoral thesis at Lancaster University (Papageorgiou, 2007) and a research project at Trinity College London

• Task: sort descriptors into the six levels



Data collection (2)Descriptors N Number of judges per administration Ratings

Sept 2005 1st

Sept 2005 2nd

November 2005

February 2006

July 2006

Speaking 30 12 12 10 11 - 1350

Writing 25 12 12 10 11 - 1125

Listening 19 12 12 10 11 - 855

Reading 20 12 12 10 11 11 1120

Global 30 12 12 10 11 - 1350

Total 124 5800



Data analysis • Analysis: FACETS Rasch computer program• 3 facets: descriptors-raters-occasions• Rank-ordering of elements of facets on a common

scale• Fit statistics (Bond and Fox, 2001; McNamara,

1996) Overfit: too predictable pattern Misfit: more than expected variance• Acceptable range of fit statistics Descriptors: .4-1.2 (Linacre & Wright, 1994) Raters: .5-1.5 (Weigle, 1998)



Results: Writing Levels A1-B1+ -2 + W11 B1 + + 3 + + + 3 + + 3 + + 3 + 3 + 3 + 3 + 3 +| | W15 B1 | | | 3 | | | | | | | | | | |+ -3 + + + + + --- + + + + + + + + + +| | | | | | | | | | | | | | | |+ -4 + W12 B1 + + + + + + + + --- + --- + --- + + + --- +| | W16 B1 | | --- | | | --- | --- | | | | | --- | --- | |+ -5 + + + + --- + + + + --- + + + 2 + + + +| | | | | | | | | | | | | | | 2 |+ -6 + + + 2 + + 2 + + + 2 + 2 + + + + 2 + +| | W4 A2 | | | 2 | | | 2 | | | 2 | --- | 2 | | |+ -7 + + + + + + 2 + + + + + + + + +| | | | | --- | | | | --- | | | | | --- | --- |+ -8 + W19 A2 W6 A2 + + --- + + + + + + + + + + + +| | | | | | | | --- | | --- | --- | | --- | | |+ -9 + + + + + + --- + + + + + + + + +| | | | | | --- | | | | | | | | | |+ -10 + W24 A1 + + + + + + + + + + + + + +| | W25 A1 | | | | | | | | | | | | | |



Results: Writing Levels B2-C2+ 7 + + + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) +| | W18 C2 W23 C2 | | | | | 5 | | | --- | | --- | 5 | | --- |+ 6 + W1 C2 W10 C2 W13 C1 W20 C1 W3 C2 W9 C2 + + 5 + 5 + + + 5 + 5 + + 5 + + + 5 + +| | W14 C2 W2 C1 W21 C1 | | | | 5 | --- | | | 5 | | | --- | | 5 |+ 5 + + + --- + --- + + + --- + --- + + --- + 5 + + + +| | W17 C1 | | | | --- | | | | --- | | | | --- | --- |+ 4 + + + + + + + + + + + --- + + + +| | W5 B2 | | | | | | | | | | | | | |+ 3 + W22 B2 W7 B2 + + + 4 + 4 + 4 + 4 + 4 + 4 + 4 + + 4 + + +| | | | 4 | | | | | | | | | | 4 | 4 |+ 2 + + + + + + + + + + + 4 + + + +| | | Claudia 3 | | | | | | | | | | | | |+ 1 + + + + --- + --- + + --- + + --- + + + + + +| | | Matt 8 | | | | --- | | --- | | --- | | | --- | |* 0 * * Alice 1 Andrew 6 Kate 2 Lora 7 Roseanne 11 Sally 12 Tim 9 * --- * * * * * * * * --- * --- * * --- *| | W8 B2 | George 5 Rita 4 | | | | | | | | | | | | |+ -1 + + Nicola 10 + + + 3 + + + + + + + + + +| | | | | | | | 3 | | 3 | | | | | |



Results: Raters+ 7 + + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) +| | | | | | 5 | | | --- | | --- | 5 | | --- |+ 6 + + 5 + 5 + + + 5 + 5 + + 5 + + + 5 + +| | | | | 5 | --- | | | 5 | | | --- | | 5 |+ 5 + + --- + --- + + + --- + --- + + --- + 5 + + + +| | | | | --- | | | | --- | | | | --- | --- |+ 4 + + + + + + + + + + --- + + + +| | | | | | | | | | | | | | |+ 3 + + + 4 + 4 + 4 + 4 + 4 + 4 + 4 + + 4 + + +| | | 4 | | | | | | | | | | 4 | 4 |+ 2 + + + + + + + + + + 4 + + + +| | Claudia | | | | | | | | | | | | |+ 1 + + + --- + --- + + --- + + --- + + + + + +| | Matt | | | | --- | | --- | | --- | | | --- | |* 0 * Alice Andrew Kate Lora Roseanne Sally Tim * --- * * * * * * * * --- * --- * * --- *| | George Rita | | | | | | | | | | | | |+ -1 + Nicola + + + 3 + + + + + + + + + +| | | | | | | 3 | | 3 | | | | | |



Results: Occassions+ 7 + + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) + (6) +| | | | | | 5 | | | --- | | --- | 5 | | --- |+ 6 + + 5 + 5 + + + 5 + 5 + + 5 + + + 5 + +| | | | | 5 | --- | | | 5 | | | --- | | 5 |+ 5 + + --- + --- + + + --- + --- + + --- + 5 + + + +| | | | | --- | | | | --- | | | | --- | --- |+ 4 + + + + + + + + + + --- + + + +| | | | | | | | | | | | | | |+ 3 + + + 4 + 4 + 4 + 4 + 4 + 4 + 4 + + 4 + + +| | | 4 | | | | | | | | | | 4 | 4 |+ 2 + + + + + + + + + + 4 + + + +| | | | | | | | | | | | | | |+ 1 + + + --- + --- + + --- + + --- + + + + + +| | | | | | --- | | --- | | --- | | | --- | |* 0 * Feb 06 Nov 05 Sept 05 1st Sept 05 2nd 1 Andrew 6 Kate 2 Lora 7 Roseanne 11 Sally 12 Tim 9 * --- * * * * * * * * --- * --- * * --- *| | | George 5 Rita 4 | | | | | | | | | | | | |+ -1 + + Nicola 10 + + + 3 + + + + + + + + + +| | | | | | | | 3 | | 3 | | | | | |



Results: Correlations

Descriptors Spearman

Speaking .959

Writing .946

Listening .968

Reading .975

Global .980

Correlations of scaling between the judges and the CEFR volume



Summary of results • Trained judges perceived language ability as

intended in the CEFR• Almost identical scaling

• Cut-offs between B2-C1 and C1-C2 unclear

• Competences other than linguistic: misfitting descriptors

• Unclear and inconsistent wording resulted in level misplacement by the judges

• Mixed effect of training



Implications of findings• Common understanding of the construct in the

CEFR scales can be achieved, but

• How valid is it to claim that a test is linked to B2 instead of C1 and C1 instead of C2?

• How can sociolinguistic and strategic competences be tested in relation to the CEFR?

• Can SLA research help better understand these issues?



Contact details

Spiros Papageorgiou

University of Michigan English Language Institute

500 East Washington Street

Ann Arbor, MI

48104-2028

USA

[email protected]

university of michigan english language institute testing and certification division behavioural...

Documents

certification division

language examinations

cefr data

cefr scales

cefr rank

cefr fulcher

data analysis analysis

writing levels a1b1