Raili HildénRaili Hildén University of Helsinki University of Helsinki

Relating the Finnish School Relating the Finnish School Scale to the CEFRScale to the CEFR

Starting pointStarting point

• ““The construction of a comprehensive, The construction of a comprehensive, transparent and coherent framework transparent and coherent framework for language learning and teaching for language learning and teaching does not imply the imposition of one does not imply the imposition of one single uniform system. On the contrary, single uniform system. On the contrary, the framework should be open and the framework should be open and flexible, so that it can be applied, with flexible, so that it can be applied, with such adaptations as prove necessary, such adaptations as prove necessary, to particular situations.” (CEFR, 2001, to particular situations.” (CEFR, 2001, p. 7)p. 7)

How was the Finnish school scale (FSS) designed?

• Decision to adapt/adopt the CEFR levels was made by the language experts invited to be members of a national curriculum development working group; approved by the supervising board

• Sources consulted: • CEFR – scales for multiple communicative activities• Canadian Benchmarks • Teacher judgement • Working group members commented on the drafts

How was the school scale designed?

• Several versions were produced based on internal feedback

• First round of empirical check for inter- rater consistency (by the team members)

• Re-formulation of descriptors with low agreement/consistency

• Second round of empirical validation

• Agreeing on the current formulations

Proficiency levels (and their labels) in Finnish language curricula

http://www.oph.fiA1 Basic communication in the most familiar situations

A1.1 First stage of elementary proficiency

A1.2 Developing elementary proficiency

A1.3 Functional elementary proficiency

A2 Communication in basic social situations and simple description

A2.1 First stage of basic proficiency

A2.2 Developing basic proficiency

B1 Communication in everyday life

B1.1 Functional basic proficiency

B1.2 Fluent basic proficiency

B2 Coping with regular relationships with native speakers

B2.1 First stage of independent proficiency

B2.2 Functional independent proficiency

C1-C2 Proficient language use in demanding contexts

C1.1 First stage of skilled proficiency

Categories included in the Finnish language curricula

Listening comprehension• Themes, text and tasks

(1)• Conditions and

constraints (2)

Speaking • Themes, texts and tasks

(monologue and interaction)

• Fluency

• Pronunciation

• Linguistic range

• Linguistic control

Reading comprehension• Themes, texts and

tasks (2)• Conditions and

constraints (1)

Writing • Themes, texts and

tasks (2)• Linguistic range

• Linguistic control

Research Questions

RQ1. What is the level of agreement between judges on the CEFR level of the FSS descriptors?

RQ2. How are the Finnish syllabus descriptors related to CEFR levels?

Design

• FSS level descriptors were split up into 184 statements

• Listening 38, Speaking 66, Reading 31 Writing 49 statements

• The statements were coded and grouped in terms of communicative activities

• Criterion scales used in the rating of FSS descriptors were selected from among relevant CEF scales

Design

• A sample of 40 Finnish language teaching experts were contacted by an e-mail questionnaire

• 20 experts returned the questionnaire• A randomised selection of statements

referring to each of the four communicative activities was e-mailed to the raters (in Finnish translation by Huttunen & Jaakkola 2003)

Conducting the research

Listening comprehensiona (38 descriptors)

Raters

Sent to:Returned

by:

1. Themes and textsb (13) 40 20

2. Tasks and activities (14) 16 8

3. Conditions and constraints (11) 15 10

Notes:a/ CEFR scale used for the rating task- Overall listening comprehensionb/ Anchor descriptors, rated by all raters

Speaking (66 descriptors)

CEFR scale Raters

Sent to: Returned by:

1. Texts, themes and tasks (21)

Overall spoken interaction

40 20

2. Fluency (10) Spoken fluency 14 6

3. Pronun-ciation (10)

Phonological control

15 8

4.Linguistic range (14)

Vocabulary range 15 9

5. Linguistic control (11)

Grammatical accuracy

15 10

Conducting the researchReading comprehensiona (31 descriptors)

Raters

Sent to: Returned by:

1. Themes and textsb (10) 40 20

2. Tasks (11) a 14 6

3. Conditions and constraints (10)14 7

4. Tasks and activities c 12 7

Notes:a/ CEFR scale - Overall reading comprehensionb/ Anchor descriptors, rated by all ratersc/ CEFR scale Reading for information and argument

Writing (49 descriptors)

CEF scale Raters

Sent to: Returned by:

1. Texts and themes (13)

Creative writing 40 20

2. Tasks and activities (12)

Overall written interaction

15 10

3. Linguistic range (11)

General linguistic range

15 7

4. Linguistic control (13)

Grammatical accuracy

15 7

5. Texts and themes (12)

Overall written production

15 6

FSS level codes CEFR level codes

A11 - 1

A12 - 2

A13 - 3

A21 - 4

A22 - 5

B11 - 6

B12 - 7

B21 - 8

B22 - 9

C11 -10

A1 - 1

A2 - 2

A2+ - 3

B1 - 4

B1 + - 5

B2 - 6

B2+ - 7

C1 - 8

C2 - 9

RQ1.The range of raters’ agreement

Frequency Percent Valid Percent Cumulative

Percent 0 10 5,4 5,4 5,4 1 40 21,7 21,7 27,2 2 42 22,8 22,8 50,0 3 39 21,2 21,2 71,2 4 26 14,1 14,1 85,3 5 21 11,4 11,4 96,7 6 3 1,6 1,6 98,4 7 2 1,1 1,1 99,5 8 1 ,5 ,5 100,0

Valid

Total 184 100,0 100,0

Range

Note: Ranges of 7 and 8 were checked after the presentationand detected to be due to clerical errors. Thus the ”true” range isFrom 0 to 6.

variance=2,37 FINSS=B21 CEF=6 CEF reference B2

R12

1 5,0 5,0 5,0

7 35,0 35,0 40,0

5 25,0 25,0 65,0

7 35,0 35,0 100,0

20 100,0 100,0

4,00

6,00

8,00

9,00

Total

ValidFrequency Percent Valid Percent

CumulativePercent

RQ 1.Agreement with the CEF levelsExample of a descriptor with a low level of agreement between raters.“Can identify the writer’s bias and the purpose of the text and locate and integrate several specific pieces of information in a longer text. Can quickly identify the content and relevance of new items deciding whether closer study is worthwhile”

Level

variance=0 FINSS=A11 CEF=1 CEF reference A1

RQ 1.Agreement with the CEF levels Example of a descriptor with a high level of agreement between raters. “Can write the alphabet of the language and all numbers and numerals. Can write down basic personal identification information and write a small number of familiar words and simple phrases.”

W42

11 55,0 100,0 100,0

9 45,0

20 100,0

1,00Valid

SystemMissing

Total

Frequency Percent Valid PercentCumulative

Percent

RQ2. Agreement between syllabus level descriptors (syllcode) and the

original CEFR levels (levcode)

levcode

A1 A2 B1 B2 C1 C2 Total

A1 28 18 1 0 0 0 47 A2 2 25 8 0 0 0 35 B1 0 2 26 6 0 0 34 B2 0 0 4 28 12 1 45

syllcode

C1 0 0 0 11 11 1 23 Total 30 45 39 45 23 2 184

Note: A quite good level of agreement was oberved (65%). There is, however, some tendency for an overestimation: 19 descriptors (10%) were assigned to a lower level while 46 (25%) were assigned to higher level.

RQ1. Range distribution per skill

WSRL

skill

8

7

6

5

4

3

2

1

0

range

The average range is 2.66 for the whole pool. The range distribution per skill can be seen in the boxplot above, showing no clear differences between the four skills.

RQ2. Absolute agreement per skill between syllabus level and CEF level

RQ1. Factors affecting the ratings (= raters are quite ”homogeneous”)

RQ2. Agreement between individual rating and original (initial) levels

(Syllabus - syll & CEF - level)

Plans for further exploration

• Calibrating the FSS descriptors• Exploring the link to the Canadian

Benchmarks in more detail• Re-formulating or removing problematic

descriptors• Empirical validation and exemplication of

the FSS scales through benchmarks for comprehension tasks and for oral and written performance samples

Summary

• The correspondence between the new Finnish school scale (FSS) and the CEFR scales were studied.

• Research question 1: What is the level of agreement between judges on the CEFR level of the FSS descriptors?

• Research question 2: How are the Finnish syllabus descriptors related to CEFR levels?

• 20 experienced raters judged FSS descriptors using relevant CEFR scales

• A good agreement was reached: 65% of the FSS descriptors were assigned to the original CEFR levels. For the rest of the cdescriptorts, some tendency of overestimation was observed.

• Inter-rater agreement was also quite good.

Acknowledgement

I wish to thank – Sauli Takala, University of Jyväskylä– Feljanka Kaftandieva, University of Sofia

for their help in carrying out this study.