dimova: national oral language assessment in denmark: where are we now and where are we going?

Overskrift her Navn på oplægsholder Navn på KU-enhed

For at ændre ”Enhedens navn” og ”Sted og dato”: Klik i menulinjen, vælg ”Indsæt” > ”Sidehoved / Sidefod”. Indføj ”Sted og dato” i feltet for dato og ”Enhedens navn” i Sidefod

National oral language assessment in Denmark: Where are we now and where are we going?

Slobodanka Dimova

Centre for Internationalisation and Parallel

Language Use (CIP)

Centre for Internationalisation and Parallel Language Use

Tekst starter uden punktopstilling For at få punkt-opstilling på teksten, brug forøg indrykning For at få venstre-stillet tekst uden punktopstilling, brug formindsk indrykning

Overskrift her


Introduction

Overview of Danish educational system

Oral English language exams in 9th grade -- Folkeskolens Afgangsprøve (FSA)

A-level and B-level oral English exams in upper secondary school (gymnasium)

Synergies and tensions between different aspects of exam development, administration, and use

Future developments and research



Overskrift her


Overview of the Danish education system (http://eng.uvm.dk/Education/Overview-of-the-Danish-Education-System)


FSA


Overskrift her


Foreign language examination system

New national structure to promote evaluation and assessment in compulsory education

•Since 2006, creation of a new national structure to establish the importance of evaluation and assessment in compulsory education

•Three national bodies with a role in the evaluation and assessment of compulsory education: the Education Agency; the Quality and Supervision Agency; and the Danish Evaluation Institute (EVA)

•The Quality and Supervision Agency is responsible for the financial, institutional and pedagogical supervision of both compulsory and upper secondary education. Major tasks include developing and running the national tests and final examinations in compulsory education, implementation of international assessments, development of materials for schools in the Evaluation portal.



Overskrift her


FSA ORAL ENGLISH EXAM (9th grade)

The FSA oral English exam consists of two tasks:

1. Presentation on a selected topic (~5 min.)

2. Conversation/interview on specific topics (~8 min.)

Total examination time including evaluation (~20min.)

Individual exam (though pupils can prepare in pairs or groups)

Two examiners (teacher and external examiner)

Mandatory for all pupils



Overskrift her


Before the exam day

• The teacher, together with pupils, develops a bibliography with reading and video material organized around at least three main themes. Each theme consists of several different topics.

• The pupil chooses a topic under one of the themes.

• The pupil selects sources related to the chosen topic and prepares the presentation. This is done with the teacher as supervisor.

• The pupil prepares and submits an outline of his/her presentation before the exam day.

• The teacher approves the student's outline with a signature. The pupil also signs the outline.



Overskrift her


Examples of themes and topics

Themes Topics

Life in South Africa before and after the Apartheid

Life in the townships AIDS in SA Apartheid Racism Sports in SA Story/movie/poem related to the topic

Gun violence in the USA The American Constitution The Ku Klux Klan Gang life Eminem’s lyrics

Fit or fat? Fitness Children and obesity Eating disorders What’s wrong with McDonald’s? What’s healthy living?



Overskrift her


An example of presentation outline Pupil’s name and class:

Theme: Young People Today My topic: Obesity

1. Why did I choose this topic?

a) My family background. b) Films I’ve seen. c) A personal interest

2. Different kinds of eating disorders

Obesity due to binge-eating; Anorexia, Bulimia

3. Who becomes obsessive – and why?

A case story

4. Whose problem is it – and what should be done about the problem?

A personal problem

”Society is to blame”- or?

School Lunch

TV-programs

Sources: Morgan Spurlock’s film “Super Size Me”,

Jamie Oliver’s Diary : http://www.jamieoliver.com/diary/

British School Dinners:

http://www.channel4.com/life/microsites/J/jamies_school_dinners/fom/index.html ns

Student’s signature: Teacher’s signature:



Overskrift her


During the exam

Task 1: Pupils give a 5-min presentation on the selected topic.

• Internal-examiner (teacher) can ask questions related to the topic

• External-examiner (censor) can ask only clarifying questions

• Examiners can use the pupil’s presentation outline as the basis for questions during the exam

Task 2: Pupils draw one card from a number of cards containing a question and pictures related to the themes covered in the bibliography.

• Pupils should take an initiative in a conversation with the examiners



Overskrift her


An example of Task 2 card



Overskrift her



Grading criteria One overall grade based on four areas:

1.Communicative skills 2.Language use 3.Preparation (acquisition) 4.Culture and society

Teacher and censor each assign a grade, but final/reported grade is decided through discussion.


Overskrift her


After the exam

The external-examiner (censor) prepares a report discussing:

• Text selection

• Themes and topics

• Pupils’ outlines

• Sources used for preparation

• Questions in Task 2

• Common mistakes

• Examination process

• Grading

• Complaints

The report is published and used for improvement of future examination procedures.



Overskrift her


Synergies

• Dynamic collaboration among pupils, teachers, and external-examiners, involvement of all stakeholders in the process

• Direct relation to material covered in the classroom, i.e. integrated teaching, learning, and assessment

• Clear expectations about the examination tasks, i.e. examination procedure transparency and lower negative individual impact



Overskrift her


Tensions

• The exam is in line with the Common Goals outlined by the curriculum. However,

• The Common Goals (Fælles Mål) are broad and lack specific description of the assesed construct.

____________

• FSA is nationally mandated and grades are used to a certain extent for school and teacher comparison. However,

• The local administration of the exam brings a large degree of variability (e.g., topics, task 2, grading, mentorship).



Overskrift her



A-LEVEL ORAL ENGLISH EXAM (stx)

The A-level oral English exam consists of two parts:

1.Presentation based on text analysis

2.Interview about the analysis

Preparation time (60min.)

Examination time including evaluation (~30min.)

Two examiners (teacher and external examiner)

Only randomly selected pupils take it


Overskrift her


Before the exam day

• Pupils prepare throughout all three years in gymnasium by

• reading texts related to different topics

• analyzing texts

• learning different terminology used in text analysis

• Teachers prepare at least three new texts related to the topics covered in class and send them to external examiners (censors)

• Censors check the teacher selected texts for the fulfillment of the requried criteria and approve them

• Pupils are selected to take the exam by random draw



Overskrift her


During the exam: Task (text analysis)

• Pupils are asked to prepare a presentation based on a literary or non-literary text. In the presenation they should,

• Analyze the text

• Interpret the text

• Relate the text to a topic discussed during the semester

• The text specifications are as follows:

• 2-4 pages

• New (not discussed in class)

• 1300 letters (if a short story or book chapter)

• 30 lines from a poem

• 30 lines from a play



Overskrift her


Task example

Theme: Urban Life Past and Present

Text: An excerpt from the novel Saturday by Ian McEwan, Jonathan Cape 2005, p. 74-81.

The novel is set in London, 2003, and describes a day in the life of Henry Perowne, a succesful neurosurgeon. In this excerpt he is on his way to his weekly game of squash. The streets are filled with anti-war protestors. The occasion is the impending war against Iraq.

…the excerpt follows…



Overskrift her



The emphasis is on:

1. Coherent (fluent) and mostly correct English

2. Presentation, analysis, understanding, interpretation and discussion in relation to the knowledge gained in working with the topic studied

3. Communicating one’s own views and arguments


Overskrift her


Synergies

• Some collaboration among pupils, teachers, and external-examiners, involvement of all stakeholders in the process

• Direct relation to the material covered in the classroom, i.e. some integration of teaching, learning and assessment.

• Clear expectations about the examination tasks, i.e. examination procedure transparency and lower negative individual impact.

• Integration of two language skills reading and speaking.



Overskrift her


Tensions

• The exam is nationally mandated and grades are used to a certain extent for school and teacher comparison. However,

• The local administration of the exam brings a large degree of variability.

______________

• Emphasis is on literary text analysis. However,

• Grades are used for admission in EMI programs at university level.



Overskrift her


Where are we now?

• Traditional oral exams—have existed a long time with small revisions

• Practicality:

• FSA and A-level/B-level oral exams are administered locally

• No additional resources needed except for censors

• Authenticity:

• Materials in prompts are authentic

• Authentic tasks?



Overskrift her


Where are we now? • Reliability:

• inter- and intra- grader reliability?

• administration reliability?

• pupil reliability?

• Validity:

• content validity

• construct underdefined, especially for FSA

• Possible construct-irrelevant variation?

• Consequences:

• stx scores use for admission in EMI

• FSA scores used for employment

• Washback:

• too much emphasis on literary texts

• 9th grade FSA focused on topic and presentation development



Overskrift her


Where are we going?



Overskrift her


Where are we going?

OPTION 1

• Keep traditional testing methods, but restrict result interpretations to local contexts

• No generalization

• No school comparison

• Local use

• Advisory purposes



Overskrift her


Where are we going?

OPTION 2

• Improve definition of the construct

• Standardize tasks

• Standardize examination procedures

• Monitor intra- and inter-rater reliability

• Conduct on-going validation investigations

• Re-think the relationship between tasks and

expectations for real-life language use

• Follow the consequences of exam changes



Overskrift her


The right way…

…depends on many factors.

• If FSA and A-level exams become higher stakes exams, quality measures have to be implemented

• Tradition could still be maintained

• Balancing practicality and other exam qualities will still greatly depend on the available resources (financial, material, professional)



Byt billede: Ny slide og klik på ikon, indsæt billede

Thank you!

[email protected]


dimova: national oral language assessment in denmark: where are we now and where are we going?

Education

examination procedure transparency

process direct relation

examination tasks

parallel language

material covered

large degree

local administration

national structure