Language teaching and language assessment in a multilingual Europe. A

case of quality, diversity and transparency

Piet van AvermaetCentre for Language and Migration

KULeuven

Economical, cultural

and social integration

(unification?) in Europe

Language Policy Context in Europe

National interests (19th

century: one nation, one

state, one language)

Multilingual policy in

Europe

Language Policy Context in Europe: paradoxes

Monolingual policy in

different European

countries

Positive attitude in Europe towards use of more than one language

Language Policy Context in Europe: paradoxes

Negative attitude in European countries towards use of non European languages

Society handles with language diversity

in a schizophrenic way

Language Policy Context in Europe

Argue in favour of multilingualism

Demand/require the use of one language

Diversity is important and has an extra value

Unity, nation building, uniformity

Language Policy Context in Europe

Striving for monolingualism is easier to argue for

Surplus value of multlingualism is difficult to make a case for; difficult to bring up arguments

Economical,practicalVague surplus value of cultural and linguistic diversity

Language and dealing with languages is a continuous

balancing between

Language Policy Context in Europe

•Uniformity•clarity•transparency•economical way of thinking•consistency•straightforwardness

•Diversity•identity•be opposed to•react against

There will always be a tendency towards uniformization and there will always be a counter

reaction by people

Language Policy Context in Europe

Process of language evolution and language change. Languages are not static but dynamic. It is

important to have people who stress nowadays

the value of diversity

Now two major tendencies

Language Policy Context in Europe

In a climate of globalisation and economic thinking there is a tendency towards uniformization and uniformity (one language/one standard)

People and media making more and more use of different communication ‘modes’: pictures/graphs/…/ linguistic code mixing

Policy / perception / discourse Daily behaviour

In the context just described, recent developments in Europe like Bologna agreement, Barcelona indicators, the CEF and the Manual have to be situated and interpreted.

Bologna agreement, Barcelona indicators, CEF and Manual

Keywords are diversity and transparency

Often interpreted and translated in many different ways:

diversity often only at level of European official languages

transparency often as uniformity and loss of specificity

One concept people often forget to mention or it is often taken for granted

Bologna agreement, Barcelona indicators, CEF and Manual

quality in language teaching

quality in language assessment

Define for yourself: quality, diversity and transparency

Organise teaching and assesment along these lines: levels and content

Make a clear statement

Consequences of Bologna and CEF for University Language teaching Institutes

Original aims

What?

CEF as POINT OF REFERENCE

Background and role of CEF

Only the overall proficiency scale is known/used

Educational misuse

Assessment misuse

Ethical misuse

Misinterpretations, misuse and impact of CEF

Quality needs, context and functionality

reliability, validity, efficiency, impact/effect

Diversity (linguistic and language) and transparency

needs analysis

domains

description of aims/context/level

relate to CEF

Bologna - CEF and language teaching/assessment

Instruments

CEF itself

Manual*

Population/students

ALTE/COP/checklists/minimum standards*

Bologna - CEF and language teaching/assessment

Evaluation is gathering and judging (assessing) students PERFORMANCES

3 central questions:

Why?

What?

How?

Language assessment: tests and alternative assessment

Shifting on 4 axes. Position on each axis depends on answer on 3 question

TIME

INVOLVEMENT IN SCORING/MARKING

TYPE OF DATA/INFORMATION GATHERED

WAYS OF GATHERING INFORMATION/DATA

Alternative assessment

Alternative assessment is NEW and NOT NEW

NOT: we do it daily

NEW: try to make it more formal/explicit. Give it a higher status in making/taking decisions for reasons of reliability, validity and efficiency

Alternative assessment

Two examples

ALTE Code of Practice

Checklists

Minimum standards

CEF Manual to relate examinations to CEF

Quality, diversity and transparency

The ALTE Code of Practice

Aim - to guarantee fairness for the users of the examinations

Two main groups of user:

Primary users are the candidates who take the exams and whose lives are affected by the results

Secondary users include: sponsors of the candidates, teachers, parents, funding agencies, employers, language teaching institutions, etc.

The Code of Practice identifies the roles of three groups of stakeholder in the testing process:

the examination developers - e.g. members of ALTE

the examination takers - primary users - who take the examinations by choice, direction or necessity

the examination users – secondary users - who require the examination for some decision-making or other purpose

Developing the ALTE Code of Practice

The Code of Practice lays down four broad areas of responsibility:

developing examinations interpreting examination results striving for fairness informing examination takers

Developing the ALTE Code of Practice

Quality Considerations for ALTE Members

Two aspects of Quality

Better understanding of Principles of Good Practice - theoretical concerns and knowledge

The practical application of the principles within each ALTE organisation

developing better systems managing innovation and change monitoring standards

The ALTE Principles of Good Practice

Based on VRIP features Validity Reliability Impact Practicality

Plus additional feature

Quality of service

How to set ALTE Quality Standards?

Quality standards

Quality Standard

In need of improvement

Satisfactory

“Best Practice Models”

Good practicet

QMS approach:

Monitor quality through self-assessment

Seek confirmation that standards are being met

e.g. through peer review within ALTE

ALTE Quality Management System - QMS

Discuss and agree on minimum standards

But establish “best practice” models as long-term targets

Establish desired outcomes and impacts within each member organisation

Move towards good practiceAim at continuous improvement

Applying an ALTE QMS

BUT each ALTE organisation is different !

Organisation

ExamsUnit

External regulation &

fundinge.g. by Govt.

ALTE

Admin.

DecisionMaking

Unit

ALTE member institution(Internal stakeholders)

External collaboratorse.g. Item writers,Examiners, etc

Test-taking contextExternal

stakeholders

USERLearners &test-takers

Schools USERUSERUSERS

External regulation &

fundinge.g. Govt.

Test centres

Adopt the QMS approach and the revised Code of Practice checklists

Apply the checklists to the 4 aspects of the Test Development and Administration Cycle

Test Design and Construction

Administration

Processing - marking, grading, issue of results

Analysis and Review

ALTE QMS

Identify within each organisation:

Current strengths

Areas in need of immediate improvement

Areas for long-term development

Jointly agree on activities to support ALTE Members in raising standards:

e.g. workshops, sub-groups projects etc.

ALTE QMS

Manual

Preliminary Pilot Version

Relating Language Examinations to the CEF

What the brief was…..

Provide guidelines and suggest procedures to

facilitate a common understanding of the CEF

levels.

Project Timeline

July 2002 Helsinki seminar

September Project approval

November Authoring group starts work

Nov-January 2003 Consultation on structure

Jan-June Consultation on content

June Consultants’ meeting (v.9)

September Preliminary Pilot version

January 2004 Standardised videos and benchmarked items

The Authoring Group

Coordinator: Brian North

Neus Figueras

Sauli Takala

Piet van Avermaet

Norman Verhelst

+

Sounding board Consultants

The Manual offers guidance to users to...

describe the examination coverage, administration and analysis procedures,

relate results reported from the examination to the “Common

Reference Levels” presented in Chapter 3 of the CEF,

provide supporting evidence tht reports the procedures followed to do so.

The Manual ….

provides a guide specifically focussed on procedures involved in the validation of a claim that a certain examination or test is linked to the CEF,

but.........

The Manual does not….

provide a general guide how to construct good

language tests or examinations, or

prescribe any single approach to constructing

language tests or examinations.

Structure of the pilot version

Chapter 1: Introduction

Chapter 2: The Common European Framework

Chapter 3: Familiarisation

Chapter 4: Specification

Chapter 5: Standardisation

Chapter 6: Empirical validation

Chapter 7: Guidelines for Reporting

Appendix: Report forms

To be followed by.....

Benchmarked performances

Benchmarked reading items

Reference supplement

Approach adopted

Four inter-related sets of procedures which

contribute to the validation process:

Familiarisation

Specification

Standardisation

Empirical Validation

SPECIFICATION OF EXAMINATION CONTENT

Internal familiarisation with CEF

Internal validity: description and analysis of° General examination content;° Process of test development;° Marking, grading, results;° Test analysis and post-examination review.

External validity: Relate° general examination description to CEF scale;° description of communicative activities tested to CEF scales;° description of aspects of communicative language competence tested to CEF scales.

CLAIM of link to CEF

Graphical presentation of examination(s) related to CEF levels

C2

C1

B2

B1

A2

A1

overall listening reading spoken spoken written written

interaction production interaction production

STANDARDISATION OF JUDGEMENTS

Familiarisation training

Training in assessing performance in relat -ion to CEF levels using standardised samples

Training in judging theDifficulty of test itemsin relation to CEFstandardised items

Benchmarking localperformance samples to CEF levels

Judging the difficultyof local items in relat-ion to CEF levels

Dissemination and implementation

CLAIM (reinforced on basis of specification)

EMPIRICAL VALIDATION THROUGH ANALYSIS OF TEST DATA

Data collection

Internal validation:

Confirming the psychometric quality of the test

External validation:

Confirming the relationship to the CEF through an independent measure

CLAIM (confirmation on basis of specification and standardisation)

VISUAL REPRESENTATION OF PROCEDURES TO RELATE EXAMINATIONS TO THE CEF

"BUILDING AN ARGUMENT"

THEORETICAL RATIONALE

QUALITATIVE VALIDATION PROCEDURE QUALITATIVE VALIDATION PROCEDURE QUANTATIVE VALIDATION PROCEDURE

SPECIFICATION OF EXAMINATION CONTENT

STANDARDISATION THROUGH TRAINING

EMPIRICAL VALIDATION THROUGH DATA ANALYSIS

Internal familiarisation with CEF Familiarisation training Data collection

Training in assessing performance to CEF using standardised samples

Training in judging thedifficulty of test itemsto CEF standardised items

Internal validity: description and analysis of° General examination content;° Process of test development;° Marking, grading, results;° Test analysis and post-examination review.

Internal validation:

To be completed

Benchmarking localperformance samples to CEF

Item difficulty Estimation to CEF

External validity: relate° General examination description to CEFscale;° Description of communicative activities tested to CEF scales;° Description of aspects of communicative language competence tested to CEF scales.

Dissemination and implementation

External validation:

To be completed

CLAIM of link to CEF CLAIM (reinforced CLAIM (confirmation on basis of

on basis of specification) specification and standardisation)

Recommended procedures encouragealignment of examinations to CEF atdiffering degrees of rigour appropriateto different testing contexts and to theextent permitted by resources andexpertise available.

Contributions most welcome!

Feedback, on the reading of the Manual

Piloting, on the basis of using the Manual

A Case Study, writing up the experience in using the Manual

Collecting Performance Samples for Speaking and Writing

Language teaching and language assessment in a multilingual Europe. A

case of quality, diversity and transparency

Piet van AvermaetCentre for Language and Migration

KULeuven