assessment as a washback tool: is it beneficial or harmful? nick saville director, research and...

Assessment as a washback tool:

is it beneficial or harmful?Nick Saville

Director, Research and ValidationUniversity of Cambridge

ESOL Examinations

October 2008

2Nick Saville, Cambridge ESOL

Outline

1. Introduction

2. Washback and impact: some definitions

3. Impact and the law of unintended

consequences

4. Researching impact: the Cambridge ESOL

approach

5. Conclusion


1. Introduction

• Examinations• access to opportunity

• fairness

• Exert a powerful influence on educational processes• possibly negative?

• concern for social consequences, ethicality, accountability


Questions to be addressed:

• How can educators and assessment providers foster beneficial effects from their tests and examinations?

• How can harmful consequences be avoided or their impact mitigated?


“Impact by design”

• Integral part of a framework for developing and validating examination systems

• A concept akin to social impact assessment


2. Washback and impact: definitions

• Washback (or backwash) has been broadly defined in the assessment literature as the effect of testing on teaching and learning

• One aspect of the broader phenomenon known as impact – as we shall see later


Washback

• Alderson and Wall, 1993:• 15 washback hypotheses • Based on who or what might be affected:

• Teaching• Learning • Content• Rate of learning• Sequence of teaching/learning• Degree/depth of curriculum coverage• Attitudes of teachers/learners• Etc.


Washback

• A continuum - stretching from harmful at one end, through neutral to beneficial at the other end

Negative Neutral Positive

- +


Washback

• Negative?• Restriction of content – narrowing of curriculum• Too much time practising for the test

• Positive?• Transparent objectives and outcomes• Increased motivation of learners• Increased accountability of teachers (?)

• BUT – cause and effect explanations are rarely adequate …..


Washback Models

In the language testing literature:

• Bailey (1996)

• Watanabe (2004)

• Cheng (2004, 2005)

• Green (2007)


Bailey’s Model (1996) - based on Hughes,1993

3 Ps:

Participants• students• teachers

Processes

Products• learning• teaching• materials• curricula


Watanabe’s Model (2004)

• Five dimensions:

• Specificity

• Intensity

• Length

• Intentionality

• Value

• Factors influencing the process of washback are

related to:• the test itself

• status

• stakeholders


FocalConstruct

Test designcharacteristics

item formatcontent

complexityetc.

Overlap

Potential fornegative backwash

Potential forpositive backwash

Perception oftest importance

Perception oftest difficulty

Backwash toparticipant

Important

Unimportant

No backwash

Intense backwash

Easy

Unachievable

Challenging

Washback direction

Washback intensity

Washback variability

Participant characteristics and values Knowledge/ understanding of test demands Resources to meet test demands Acceptance of test demands

Other stakeholdersCourse providersMaterials writers

PublishersTeachersLearners

Green’s Model See: Studies in Language Testing, 25,

2007: IELTS Washback in Context

Washback will be most intense –

have the most powerful effects

on teaching and learning

behaviours – where participants

see the test as challenging and

the results as important (high

stakes)

SEE BLUE ARROW


Impact

Impact concerns the effects and consequences a test can have beyond the classroom and immediate learning context:

• On individual career or life chances

• In educational systems and in society


Impact

• Impact deals with wider influences and includes the “macro contexts” - tests and examinations in society

• Washback is an aspect of impact related to the “micro contexts” of the classroom and the school


• Investigating impact is integral to validation

• an essential component in establishing the usefulness of an

assessment system - fitness for specific purposes and

contexts of use

• Consistent with Messick views of validity (1989, 1996)• Consequential aspects of validity

• fairness and ethics

Impact


• Educational systems – complex phenomena

• Stakeholder constituencies • many different stakeholders in educational processes

• complex network of relationships

Impact


LearnersTeachersTest writers/examiners Receiving institutionsSchool ownersFuture employersGovernment agenciesProfessional bodiesTest centre administratorsMaterials writersPublishersetc

Learners Parents/carersTeachersReceiving institutions EmployersSchool ownersExaminersGovernment agenciesProfessional bodiesAcademic researchersTest writers/Examinersetc

Test constructs

Test format

Test conditions

Test assessment

criteria

Test scores

Stakeholders in the testing constituency

Testing System

Contexts of test use - consequencesInputs to test design




Test constructs

Test format

Test conditions

Test assessment

criteria

Test scores

Testing System




Impact

• Multiple voices

• views, beliefs and attitudes

• Important for examination providers • to collect feedback from the stakeholder• to take stakeholder perspectives into account

• Dynamic relationship between micro and macro contexts


3. The “law” of unintended consequences

• “Any purposeful action will produce some unintended consequences” or side-effects

• “Goodhart’s Law” (or “Campbell’s Law” in the USA)• a variant of the “law” of unintended consequences


“Goodhart’s Law”

• “All performance indicators lose their meaning when adopted as policy targets”

• Examples:• England - school achievement targets - school league tables

• USA – No Child Left Behind (NCLB)

• The clearer you are about what you want, the more likely you are to get it – but the less likely it is to mean what you wanted it to! (Dylan Wiliam, 2008)


Perverse incentives?

• Assessment policy can create a tension between

• educational objectives at the micro level (teaching and learning in schools) and

• a requirement for accountability at the macro level


What role can examination providers play?

How can examination providers ensure that :

• examination systems are “fit for purpose”?

• research is carried out to “find out what is going on” in contexts of use?

• mitigating action is are carried out if/when negative effects and consequences occur?


4. Researching impact

• Cambridge ESOL

• an international examinations board

• Maxims of Test Impact

• An attempt to integrate an action-oriented approach to investigating impact into working practices

• Milanovic and Saville, 1996


Maxims for achieving/monitoring impact

Maxim 1 PLANUse a rational and explicit approach to test development

Maxim 2 SUPPORTSupport stakeholders in the testing process

Maxim 3 COMMUNICATEProvide comprehensive, useful and transparent information

Maxim 4 MONITOR and EVALUATECollect all relevant data and analyse as required.

Milanovic and Saville, 1996 - Considering the impact of the Cambridge examinations


Maxim 1 PlanUse a rational and explicit approach to test development

• a process model – cyclical and iterative

• creates the necessary conditions for appropriate tests to be developed and for on-going validation to take place

• begins with the purpose - including anticipating • how the test should (or might) be used

• how relevant and useful it is likely to be

- social consequences and value implications

• potential (unplanned) side-effects


Maxim 2 Support

Support stakeholders in the testing process

• Involvement of stakeholders - during test design and development• consultation on specifications/syllabus design• participation in pilot tests

• Professional support programmes• for institutions and individual teachers/students etc.

• Training of suitable personnel to work on all aspects of the examination cycle• item writers, examiners, etc.




Test constructs

Test format

Test conditions

Test assessment

criteria

Test scores


Testing System



Maxim 3 Communicate

Provide comprehensive, useful and transparent information

• Explain issues related to assessment adapted for the different stakeholders • the nature of the language constructs being assessed

• the meaning of language test results

• etc.

• A major challenge for all test providers!


Maxim 4 Monitor and EvaluateCollect all relevant data and analyse as required

• For example, feedback from the candidates and their teachers • from local contexts where the test is used (i.e. at the micro level)

• A long-term endeavour as it involves:• the development of suitable instruments for the collection of adequate

data• appropriate research methodologies

e.g. mixed method designs, case studies etc.

• Evaluate the test’s usefulness routinely• determine the need for changes and periodic revisions


Outcomes of impact studies

• The World-wide survey of the impact of IELTS

• The Italian Progetto Lingue 2000 (PL2000 project)

• See Hawkey, (2006) • Studies in Language Testing, 24

The theory and practice of impact studies: Messages from studies of the IELTS test and Progetto Lingue 2000


Progetto Lingue 2000

• The policy was

• “.... to introduce innovation into the teaching and learning of other

languages by putting greater emphasis on the development of

communicative competence in all grades of the school system”

• It included:

• the adoption of the Council of Europe’s Common European Framework

of Reference (CEFR) as the basis for learning objectives and standards

• certification of proficiency – the testing

• (by a certificating body recognised internationally)


Italy - PL2000 Impact Project

Main interdependent language programme stakeholders and dimensions

Learning

goals,

curriculum,

syllabus

Students

Parents

Teachers

Teacher-trainers

Curriculum developers

Testers

Publishers

Receiving institutions

Employers

Students

Parents

Teachers

Teacher-trainers

Curriculum developers

Testers

Publishers

Receiving institutions

Employers

Materials

Methodology Teacher Support

Testing


5. Conclusion• Impact research now reflects the growing importance of

evidence-based approaches to educational planning

and evaluation

• By adopting an “impact by design” approach - through

careful test development and validation strategies, and

by using the findings from impact research to guide

future actions - more effective assessment policies

and practices can be developed to meet the needs of

contemporary education


Thank you

assessment as a washback tool: is it beneficial or harmful? nick saville director, research and...

Documents

cambridge esol impact

cambridge esol washback

cambridge esol approach

cambridge esol questions

nick saville director

cambridge esol watanabes

cambridge esol greens

cambridge esol baileys