thom kiddle: item development, the cefr and the perils of cinderella testing

Eaquals International Conference, 16 – 18 April 2015

Item development, the CEFR, and the perils of Cinderella testing

Thom Kiddle

Director

NILE (Norwich Institute for Language Education)

www.eaquals.org

2 Eaquals International Conference, 16 – 18 April 2015

• Constructs and cognition

• Development of an item type • The role of the CEFR • Cinderella testing


Constructs and cognition “It is important to emphasise that constructs are not psychologically real entities that exist in our heads. Rather, they are abstractions that we define for a specific assessment purpose.” (Alderson, 2000: 118)


Constructs and cognition


Constructs and cognition “Constructs come from a theory of reading, and they are realised through the texts we select, the tasks we require readers to perform, the understandings they exhibit and the inferences we make from those understandings, typically as reflected in scores.” (Alderson, 2000: 117)


• Understanding text structure in semi-authentic reading passages (EFL exam in European context)

Original item type:

Development of an item type


• Suggested variation on item type

Intra-text-banked gap-fill

flowerpot

earth

press

sunny

garden


Potential/proposed advantages • Principled selection of lexical and

grammatical elements contributing to text structure / cohesion cf. M/C cloze vs. open cloze



M/C cloze – lexical focus


Open cloze – grammatical focus


Potential/proposed advantages • Principled selection of lexical and

grammatical elements contributing to text structure cf. M/C cloze vs. open cloze

• Banking within text allows deduction of meaning of unfamiliar lexis in context; even of target items

• Reflects and highlights role of repetition in text cohesion

• Potential diagnostic value and positive washback

• Test economy cf. Summary items



Summary item – reading comprehension


Quantitative and qualitative analysis • Encouraging facility and discrimination

indices (CTT)



48.62

58.26

58.72

0.62

0.59

0.65

Item facility and discrimination values



indices (CTT) • Repeated verbal protocol analyses

reveal differences in approach (and potential item unfamiliarity issues)



• Stronger students* treat as open cloze: predict missing word, then scan text for confirmation

• Weaker students* read whole passage for general comprehension, then attempt gaps in order.

• Strongest students* ignore instructions and fill in own words!

Verbal protocol analyses

* On external proficiency measure, not on item or test.



indices (CTT) • Repeated verbal protocol analyses

reveal differences in approach (and potential item unfamiliarity issues)

• Item/text interaction analyses • Text difficulty analysis



Text difficulty

• Text readability measures (e.g. www.readable.com)


Text difficulty


• Lexical frequency measures (e.g. Vocabprofiler at www.lextutor.ca)


Text difficulty


• Lexical frequency measures (e.g. Vocabprofiler at www.lextutor.ca)

• CEFR calculator tool (www.cefestim.ecml.at)


The role of the CEFR

• Desire for exams and tests to be aligned to CEFR

• Implications for standard-setting (IDM / Bookmark / Basket / Modified Angoff)

• Do CEFR descriptive scales address processes involved in achieving outcomes (the horizontal dimension)? Are these processes language specific?


The CEFR: assessing language processing

Council of Europe (2001: 91-2)


• Desire for exams and tests to be aligned to CEFR

• Implications for standard-setting on item / task difficulty (IDM / Bookmark / Basket / Modified Angoff)

• Do CEFR descriptive scales address processes involved in achieving outcomes (the horizontal dimension)? Are these processes language specific?

• What are the implications for scale creation of

item / task types targeting processes which may emerge at a specific level of proficiency (the vertical dimension)?

The role of the CEFR


Reading: Can read straightforward factual texts on subjects

related to his/her field and interest with a satisfactory level of

comprehension.

Reading: Can understand in detail lengthy, complex texts,

whether or not they relate to his/her own area of speciality.

The CEFR: the vertical dimension


… it may be an essential part of the construct to recognise that certain reading processes emerge and are mastered at certain levels of proficiency, and may not be pan-linguistic across (European) languages.


Cinderella testing

• What is happening in this still from an animated fairy tale?

• The worst test ever?

• Potential pitfall of desire for CEFR alignment to dictate foci of item types to only those reflected in CEFR illustrative scales?


References Alderson, J.C. (2000) Assessing Reading Cambridge: Cambridge University Press Council of Europe (2001) Common European Framework of Reference for Languages: Learning, Teaching, Assessment Cambridge: Cambridge University Press