descriptor development (north 2000) updating and extending ...• relation to theory through...

2016/3/12

1

Updating and extending the CEFR Descriptors:

Brian North

Descriptor Development (North 2000)

Intuitive Phase:§ Creating a pool of classified, edited descriptors

Qualitative Phase:§ Analysis of teachers discussing proficiency§ 32 teacher workshops sorting descriptors into categories

Quantitative Phase:§ Teacher assessment of their learners at end year

assessment point on descriptor-checklists (circa 300 teachers, 2800 learners)

§ Teacher assessment of videos of some of the same learners to further link the data set

Interpretative Phase:§ Setting “cut-points” between the common reference levels

Validity Claim

Developed scientifically:• comprehensive documentation of existing descriptions

• relation to theory through descriptive scheme • positive, independent criterion-descriptors• checking teachers could use categories & descriptors• data from real, end-of-year assessment• four educational sectors in a multi-lingual environment• three foreign languages (English, French, German)• values replicated in follow-up studies:

Basel self-assessment 0.99: ALTE 0.97; DIALANG: 0.92 / 0.96; Pearson 0.97

Communicative Activities

CommunicativeStrategies

CommunicativeLanguage Competencies

Reception Production

Understandinga native speaker

Conversation

InformalDiscussion

FormalDiscussion

Obtaining Goodsand Services

Interviewing &being interviewed

Spoken Written

Interaction Mediation

Overall Language Proficiency

CommunicativeActivities



Reception Production

Understandinga native speaker

Conversation

InformalDiscussion

FormalDiscussion

Obtaining Goodsand Services

Interviewing &being interviewed

Spoken Written

Interaction Mediation

Overall Language Proficiency


2016/3/12

2

Aspects of Competence


GeneralLinguistic

VocabularyRange

Range

GrammaticalAccuracy

PhonologicalControl

VocabularyControl

OrthographicControl

Control

Linguistic Sociolinguistic Pragmatic



Overall language Proficiency


GeneralLinguistic

VocabularyRange

Range

GrammaticalAccuracy

GrammaticalAccuracy

PhonologicalControl

PhonologicalControl

VocabularyControl

VocabularyControl

OrthographicControl

OrthographicControl

OrthographicControl

ControlControl

Linguistic Sociolinguistic Pragmatic



Overall language Proficiency

Cut-off Range on scaleAbove C2 5.10 (no descriptors)

C2 3.90 1.20

C1 2.80 1.10

B2+ 1.74 1.06

B2 0.72 1.02

B1+ -0.26 0.98

B1 -1.23 0.97

A2+ -2.21 0.98

A2 -3.23 1.02

A1 -4.29 1.06

Pre- A1 -5.39 1.10

Cut-off Range on scale

C2 3.90 1.20

C1 2.80 1.10

B2+ 1.74 1.06

B2 0.72 1.02

B1+ -0.26 0.98

B1 -1.23 0.97

A2+ -2.21 0.98

A2 -3.23 1.02

A1 -4.29 1.06

Follow up projects - Calibrated

§ ALTE ‘Can Dos’ 1991

§ Finnish AMMKIA 2009 (?)§ Swiss IEF/Lingualevel 2009§ CEFR-J 2010§ English Profile - C levels 2011§ Pearson GSE 2012-4

2016/3/12

3

Anchor items

10

20

30

40

50

60

70

80

90

-5 -4 -3 -2 -1 0 1 2 3 4 5Original calibrations (North 2000)

Pear

son

ratin

gs 2

012

Pearson 2012

100 new descriptors.

20 CEFR anchors show a high level of agreement with the original calibrations

After removing the two outliers correlation is

0.97

Extending the bank Updating and extending the 2001 scales

§ Year 1: Updating CEFR 2001 scales(2013–14)

§ Year 2: Extending with 27 new scales(2014–16)

§ Consultation: Internal and external(2016–17)

Year 1: Updating the 2001 scales

Adapting validated descriptors:§ Addition of Pre-A1§ Enriching description at C levels

§ Filling out the «plus levels»§ One entirely new scale: «Reading for Pleasure»

Authoring team (Eurocentres)

Sounding board of experts

Consultation group

Year 2: Developing new descriptor scales

Formulating & validating descriptors for new areas:§ Mediation§ Online interaction§ Reactions to Literature & art§ Plurilingual & pluricultural competences

2016/3/12

4

Mediation in the CEFR 2001

§ Language use is organized into four modes instead of four skills.

§ “Communication is an integral part of tasks where participants engage in § interaction§ production § reception§ mediation or a combination of two or more of these” (CEFR, p.157)

13


“In mediating activities, the language user is not concerned to express his/her own meanings, but simply to act as an intermediary between interlocutors who are unable to understand each other directly, normally (but not exclusively) speakers of different languages. …” (CEFR Section 4.4.4)


“Translation or interpretation, a paraphrase, summary or record, provides for a third party a (re)formulation of a source text to which this third party does not have direct access. Mediation language activities, (re)processing an existing text, occupy an important place in the normal linguistic functioning of our societies.” (CEFR, Section 2.1.3)


= Conveying received meaning§ (re)processing an existing text§ acting as an intermediary § in same language or across languages

2016/3/12

5

Towards a richer ‘model’ of Mediation

any procedure, arrangement or action designed to reduce the distance between two (or more) poles of otherness. (Coste & Cavalli, 2015)

The mediator facilitates - access to knowledge, to the grasping of

concepts- reduction of affective blocks / tensions, - building bridges towards the new, the other

17 18

Linguistic mediation

Cultural mediation

Textual mediation

Mediation through media

Social mediation

Pedagogic mediation

Pedagogic Textual

Media Social

Linguistic Cultural

19

Relational Cognitive

20

2016/3/12

6

Cognitive

21

Relational

across languages

and cultures

across worlds

across media

Cognitive

22

Relational

Plurilingual & Pluricultural

across languages

and cultures

across worlds

across media

Cognitive

LiteratureOnline

23

Relational

Categories

§ Relational Mediation § Cognitive Mediation

§ Constructing meaning

§ Conveying received meaning

§ Mediation Strategies

2016/3/12

7

Categories




§ Mediation Strategies+§ Online interaction§ Reactions to literature & art§ Plurilingual & pluricultural competences

Categories

§ Relational Mediation (5 scales)§ Cognitive Mediation

§ Constructing meaning (2 scales)

§ Conveying received meaning (9 scales)

§ Mediation Strategies (5 scales)+§ Online interaction (2 scales)§ Reactions to literature & art (2 scales)§ Plurilingual & pluricultural (2 scales)

Categories

§ Relational Mediation (5)– Establishing a positive atmosphere– Creating pluricultural space– Facilitating collaborative interaction with

peers– Managing plenary and group interaction– Resolving delicate situations and disputes

Categories

§ Cognitive Mediation§ Constructing meaning (2)– Collaborating to construct meaning– Generating conceptual talk

2016/3/12

8

Categories

§ Cognitive Mediation§ Constructing meaning– Collaborating to construct meaning– Generating conceptual talk

§ Relational Mediation– Facilitating collaborative interaction with

peers– Managing plenary and group interaction

Categories

§ Cognitive Mediation§ Constructing meaning (2)– Collaborating to construct meaning– Generating conceptual talk

Categories

§ Cognitive Mediation§ Conveying received meaning (9)– Relaying specific information – sp+wr– Explaining data (e.g. in graphs,

diagrams, charts etc.) – sp + wr– Processing text – sp + wr– Interpreting – sp only – Translation – wr only– Spoken translation of written text

(Sight translation)

Categories

§ Mediation Strategies (5)– Linking to previous knowledge– Amplifying text– Streamlining text– Breaking down complicated information– Visually representing information– Adjusting language

2016/3/12

9

Categories




§ Mediation Strategies+§ Online interaction§ Reactions to literature & art§ Plurilingual & pluricultural competences

Categories

§ Online interaction– Online conversation and discussion– Goal-oriented online transactions and

collaboration

Categories

§ Reactions to literature & art– Expressing a personal response to literature

and art– Analysis and criticism of literature and art

Categories

§ Plurilingual & pluricultural competences– Exploiting pluricultural repertoire– Plurilingual comprehension– Exploiting plurilingual repertoire

2016/3/12

10

Categories

§ Relational Mediation (5 scales)§ Cognitive Mediation

§ Constructing meaning (2 scales)

§ Conveying received meaning (9 scales)

§ Mediation Strategies (5 scales)+§ Online interaction (2 scales)§ Reactions to literature & art (2 scales)§ Plurilingual & pluricultural (2 scales)

Descriptors

Total = § 27 descriptor scales§ Circa 400 descriptors

ValidationIntuitive: Collect, classify, edit, discuss, redraftQualitative: (Phase 1)assigning to categories§ evaluating§ suggesting reformulations (shortening)

Quantitative (Rasch scaling)§ assigning to levels (Phase 2)§ Yes/No responses (Phase 3)

39

ValidationIntuitive: Team of 8Qualitative: (Phase 1)990 informants in 140 institutes Quantitative (Rasch scaling)§ Phase 2: 1294 informants in 189 institutes § Phase 3: 3503 usable responses

40

2016/3/12

11

Validation: Linked data matrix§ Descriptors appear on 2–4 (usually 3)

different questionnaires(Phases 1, 2 & 3)

= Responses linked into one data set

§ Selected CEFR descriptors appear as “anchor items” in each phase

§ Questionnaires distributed evenly, according to profiles of institutes

41

ValidationIntuitive: Collect, classify, edit, discuss, redraftQualitative: (Phase 1)990 informants in 140 institutes § assigning to categories§ evaluating§ suggesting reformulations (shortening)


42

Category + Quality

43

Descriptor 230

Can, in a training situation, spontaneously and flexibly suggest an appropriate task to help participants reflect on their existing knowledge and competences in relation to the content.

44

2016/3/12

12

Pairs: 8 ESTAB RESTR AMPL STREA ADJUS CAN'T DROP OKCoef DRCoef CLEAR % PED % REAL % Comment

265 107 ADJUS 8 100 0 5 63 4 50 5 63 Can adjust his/her language to make a complex specialist topic accessible to recipients who are not familiar with it, adapting speed and complexity of expression to the participants and avoiding idiomatic usage if necessary.

272 129 ADJUS 1 1 6 75 0 5 63 5 63 5 63 Can avoid or explain technical terminology in order to communicate with non-experts about matters within his/her field of specialisation.260 155 STREA 8 100 0 5 63 5 63 5 63 Can condense source content by excluding parts of the text that do not add relevant new information according to the needs or conventions of a given audience. 266 157 ADJUS 1 5 2 63 0 3 38 5 63 5 63 too long + mixed Can consciously adapt articulation, sentence stress, intonation, speed and volume in order to structure and highlight important aspects of the content and transitions from one topic or phase to another.264 158 ADJUS 1 5 1 1 63 13 1 13 3 38 4 50 too long + mixed Can consciously adapt virtually any linguistic or paralinguistic features of a very wide range of spoken and written texts in order to present the main content in an appropriate register and degree of sophistication and

detail to the audience concerned. 270 159 ADJUS 2 1 1 2 2 25 0 3 38 2 25 4 50 Ugh;

ADJUS+AMPL+ESTAB

Can consciously exploit intonation and paralinguistic signals , so that people can identify the communicative intention even when they did not understand all the details.

263 168 STREA 1 7 88 0 5 63 5 63 5 63 Can create a list of the most important and relevant information contained in short, everyday, informational texts (e.g., entries in brochures and guides, leaflets, posters etc) in order to pass the information on to someone else.

239 171 RESTR 1 3 3 1 13 0 2 25 3 38 5 63 complex & ESTAB + AMPL; too long; 2nd part RESTR?

Can creatively blend information from different parts of a complex text with relevant information from other sources, organizing the whole in order to both comply with the discourse conventions of the particular target genre and highlight his/her own perspective.

236 195 ESTAB 8 100 0 5 63 5 63 5 63 Can describe in a very simple manner how new information is related to something the participant(s) is/are already familiar with.258 208 STREA 1 1 1 3 1 1 38 13 2 25 4 50 5 63 Can distil information from different parts of the source text in order to make the comparisons and contrasts contained in it in the text accessible for the recipient. 257 209 STREA 1 7 88 0 3 38 3 38 3 38 Too teachery Can distil the relevant information from different parts of the source text to highlight the essential points in order to guide the recipient in drawing their conclusions x3247 215 AMPL 8 100 0 4 50 5 63 5 63 context? Can elaborate meanings contained in a source text by adding redundancy, explaining and modifying style and register in order to make it more accessible for the context and audience.246 228 AMPL 1 7 88 0 5 63 5 63 5 63 Can enhance the accessibility of complex, challenging content by explaining different aspects more explicitly and emphatically or by adding detail and subsidiary points that are lacking.259 240 STREA 2 2 3 1 25 0 4 50 4 50 5 63 STREA+ADJUS Can exclude non-relevant or repetitive information from a source text taking into consideration the linguistic limitations imposed by the intended audience.248 241 AMPL 8 100 0 5 63 5 63 5 63 Can expand information given in a form that is too condensed for the target audience, through elaboration and provision of examples.250 243 AMPL 1 6 1 75 0 4 50 4 50 5 63 ESTAB + AMPL;

too detailed; why domain (only) her?

Can expand on the content of a text on a topic of personal interest by adding background information, examples, reasoning and explanatory comments in order to make it accessible to a target audience.

243 244 AMPL 8 100 0 5 63 5 63 5 63 Can expand upon the information given in texts on complex, academic or professional topics through the inclusion of elaboration and/or examples, so that the content of the source text becomes clearer and more accessible for the target audience.

231 248 ESTAB 5 3 63 38 2 25 3 38 3 38 Can explain clearly and precisely the way something that he/she is going to introduce x3 what is going to be introduced builds on what participants are likely to already know.240 291 RESTR 4 3 1 50 13 3 38 3 38 4 50 too long + dense;

logical order is new or original?

Can focus on, and highlight in his/her target text, the salient lines of argument and the relationships between points found in different parts of a specialised source text or texts, in order to present these in a clear, concise and logical order appropriate to the particular genre.

262 292 STREA 2 6 75 0 4 50 5 63 5 63 Can focus x2 the information contained in written and/or spoken sources by eliminating repetition and digressions in order to make the essential message accessible. 251 334 AMPL 1 3 1 2 38 0 4 50 5 63 5 63 Can help people understand support understanding of unfamiliar patterns and expressions in a text, by providing additional examples that contain similar patterns and expressions.261 341 STREA 1 6 1 75 0 3 38 4 50 5 63 UGH; a bit

sophisticatedCan identify related or repeated information in different parts of a text and amalgamate x3 them in order to make the essential message salient visible/clear for the recipient.

267 357 ADJUS 1 6 1 75 0 4 50 5 63 5 63 Can interpret and restate for a listener who does not have specialist knowledge, complex, technical topics within his/her field, using suitably non-technical language as necessary.252 380 AMPL 5 3 63 0 5 63 5 63 5 63 Can make a message on an everyday topic clearer and more explicit by repeating the same meaning in different ways.273 381 ADJUS 8 100 0 5 63 5 63 5 63 Can make a specific, complex piece of information in his/her field clearer and more explicit for others by paraphrasing it in simpler language.274 382 ADJUS 7 1 88 0 3 38 5 63 5 63 Can make the main contents of a written or oral text on a subject of interest (e.g., an essay, a forum discussion, a presentation) accessible for others by paraphrasing in simpler language. 269 385 ADJUS 1 1 5 1 63 0 4 50 5 63 5 63 Can make difficult concepts in a complex text more comprehensible through spontaneous rephrasing and paraphrasing, and by providing synonyms with explanations of the slight variations and subtle distinctions

involved. 245 386 AMPL 1 5 1 1 63 0 4 50 5 63 5 63 Can make difficult content comprehensible by giving more concrete examples, summarizing explaining step by step, and repeating the main points.256 389 STREA 7 1 88 13 4 50 4 50 4 50 Can make prominent certain information in specialised or academic texts through distilling the main ideas included. 254 394 AMPL 7 1 88 13 4 50 4 50 4 50 lang comp? Can make the content of a source text more accessible by adding relevant visual information (e.g. pictures/photos) to accompany the verbal information in the text. 268 412 ADJUS 2 6 75 0 5 63 5 63 5 63 Can paraphrase the content of a text (e.g., a popular scientific article) appropriately in a different genre and register (e.g., an email to a friend) to make information more accessible.237 451 ESTAB 8 100 0 5 63 5 63 5 63 Can pose very simple questions that ask the participant(s) to think about what they know about a topic, in order to help them relate to it.241 464 RESTR 6 2 75 0 5 63 5 63 5 63 Can present information in a different way (e.g. regarding structure, order, emphasis )Can structure and emphasise information in a different way to that appearing in the source text on a topic of interest (e.g. a

specialised book) in order to comply with the conventions of the target genre (e.g. an email to a friend). 253 478 AMPL 8 100 0 5 63 5 63 5 63 Can provide repetition and redundancy in order to make new content more accessible.234 480 ESTAB 5 1 1 1 63 13 1 13 3 38 4 50 complex- simplify;

lang comp?Can raise participants’ awareness on how something he/she is going to explain builds on and extends their existing knowledge and/or skills by asking targeted questions, giving explanations, and/or providing visual representations (e.g. diagram/chart, tables, flowcharts)

238 482 RESTR 1 5 1 1 63 0 4 50 4 50 4 50 Can rearrange the introduction and development of information and arguments in a specialized text of a specific genre (e.g. an academic article) in order to comply with the discourse conventions and organisational patterns of another genre (e.g. a report, a news item, an email).

242 495 RESTR 3 2 3 38 38 0 0 2 25 2 25 example for target genre

Can represent the distinction between main points and supporting details in the source text in an appropriate way for the target genre rather than the way in which it was presented in the source text.

255 502 STREA 1 3 4 50 0 5 63 5 63 5 63 Can rewrite a source text, reorganizing it in order to focus on the points of most relevance to the planned recipients.232 512 ESTAB 8 100 0 5 63 5 63 5 63 Great! Can spontaneously pose a series of questions to encourage participants to think about their knowledge on the topic and help them to establish a personal link to what he/she is going to explain.275 577 ADJUS 7 1 88 0 4 50 5 63 5 63 Can use paraphrase to explain more simply the main points made in short, straightforward written or oral texts on familiar subjects (e.g. a short magazine article or interview) to make the contents accessible for others.

271 580 ADJUS 8 100 0 5 63 5 63 5 63 Can use rephrasing and paraphrasing to explain in a simplified, more concrete form the content of a text to a recipient.244 592 AMPL 2 3 2 1 38 13 3 38 4 50 4 50 lang comp x2? Can visually represent information (e.g. graphic organisers like mind maps, tables, flowcharts) that help participants to recognise both the key concepts and the discourse structure (e.g. problem-solution, process

stages, compare and contrast). 249 604 AMPL 6 1 1 75 13 4 50 4 50 4 50 Can, given time to anticipate problems and prepare in advance, make concepts comprehensible by giving concrete examples, summarizing step by step and repeating the main points. 233 607 ESTAB 7 1 88 13 4 50 4 50 4 50 too teacher-

orientedCan, in a training situation x2, explicitly explain the connections between the goals of the session and the personal or professional interests and experiences of the participant(s).

235 608 ESTAB 7 1 88 13 4 50 4 50 4 50 too teacher-oriented

Can, in a training situation x2, make explicit the aims of the session and show how activities relate to it.

45

Quality coefficients

46

Descriptor 232

Can spontaneously pose a series of questions to encourage people to think about their prior knowledge of an abstract issue and to help them establish a link to what is going to be explained.

47



48

2016/3/12

13

Phase 2: Assigning CEFR level

At what CEFR level do you think a person can do what is defined in the descriptor?

Pre-A1 A1 A2 A2+ B1 B1+ B2 B2+ C1 C2

49

Two types of analysis§ Classical analysis:

§ the mean and median levels rated§ the percentage that rated the intended

broader level (50% criterion)§ Rasch analysis to link the ratings to the

scale underlying CEFR levels:§ Anchoring the steps on the scale.§ Anchoring to calibrated CEFR descriptors§ Doing both of the above simultaneously § Unanchored analysis, equated

50

Percentages at each band

51

Descriptor 3COLLAB3expanding

Can build on a partner’s ideas, expanding and deepening them.

52

2016/3/12

14

Two types of Analysis§ Classical analysis:

§ the mean and median levels rated;§ the percentage that rated the intended

level;§ Rasch analysis to link the ratings to the scale

underlying CEFR levels:a) Anchoring the steps on the scale.b) Anchoring to calibrated CEFR descriptorsc) Doing both of the above d) Equating unanchored analysis

53 54

Phase 2 Rasch Analysis



55

Phase 3: Yes/No responses§ Main calibration exercise – wide circulation§ English & French

§ 192 out of c400 items calibrated in Phase 2§ 62 of these included as anchors for Phase 3§ Plus 12 CEFR anchors= 74 anchors in 365 items§ Presented again in 23 overlapping

questionnaires

56

2016/3/12

15

Phase 3: Yes/No responsesCould you, or the person concerned, do what is described in the descriptor?

57

0 Beyond my/his/her capabilities 1 Yes, under favourable circumstances 2 Yes, in normal circumstances3 Yes, even in difficult circumstances4 Clearly better than this

Replicating original CEFR descrip. research (Swiss project 1993-6: North & Schneider 1998; North 2000)

Phase 3: Yes/No responses§ Global Rasch analysis§ Separate analyses for categories less

central to construct:§ Interpretation/translation§ Online§ Literature & art§ Plurilingual/pluricultural(with the relevant anchors from Phase 2)

58

Descriptor 3COLLAB3expanding

Can build on a partner’s ideas, expanding and deepening them.

59A

im

Ph

ase

3R

asc

h

Ph

ase

2

cla

ssic

al

Ph

ase

3

Ra

sch

De

cisi

on

Log

it

SE

M

Se

ria

l

Descriptor

COLLAB03expanding

C1 B2 B2 B2 ??? 1.25 .09 3 Can build on a partner’s ideas, expanding and deepening them.

Descriptor 3

2016/3/12

16

§ Can present his/her ideas in a group and pose questions that invite reactions from other group members’ perspectives.

§ Can formulate follow-up questions to a member of a group to clarify an issue that is implicit or poorly articulated.

§ Can highlight inconsistencies in thinking, and challenge other’s ideas in the process of trying to reach a consensus.

§ Can further develop other people’s ideas and opinions.§ Can, in collaborative discussion, consider two different

sides of an issue and propose a solution or compromise.§ Can summarise the point reached at a particular stage in

a discussion and propose next steps.

B2 Descriptors for COLLAB§ Deleting c 30 good descriptors

(because of repetition/redundancy)

§ Follow up on Plurilingual/pluricultural (using Phase 2 methodology)

§ Extending descriptors for Plurilingual/-cultural

§ Developing a new Phonology scale§ Intuitive: Team of 3 + 4 consultants§ Qualitative: Phase 1: assigning to categories

and evaluating § Qualitative: Phase 3 + Phase 2 combined

Follow ups

§ Document with descriptor scales, including the rationale for each new scale

§ Report giving academic, CoE and CEFR context, account of development and validation, posing questions for consultative process

§ Archive document with all descriptors considered, plus validation history

§ Technical report for each validation phase

Documentation§ Meeting June with experts from related

Council of Europe projects § Consultation and piloting until later 2017

(following the CEFR precedent)§ Revision for publication.

The exact form of presentation and publication will be one of the issues considered in the consultation process.

Extended consultative process

2016/3/12

17

Thank you

Brian North [email protected]

descriptor development (north 2000) updating and extending ...• relation to theory through...

Documents