national conference on next generation assessments washington, 8-9 march 2010 future of assessments...

11N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

The future of AssessmentsLessons learned internationally

Washington, 9 March 2010

Andreas SchleicherHead, Indicators and Analysis Division

OECD Directorate for Education

22N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rThe future of assessments

The Holy Grail? Or the Alchemists’ Stone? Know why you are looking

You cannot improve what you cannot measure The yardstick for success is no longer

just improvement by national standards but the best performing education systems globally

Know what you are looking for A new assessment culture Responsive to changing skill requirements Capitalising on methodological advances Not sacrificing validity gains for efficiency gains

Know how you will recognise it when you find it Gauging predictive validity Impact on improving learning and teaching

Implications and lessons learned .

33N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rThe old bureaucratic system The modern enabling system

Hit and miss Universal high standards

Uniformity Embracing diversity

Provision Outcomes

Bureaucratic look-up Devolved – look outwards

Talk equity Deliver equity

Prescription Informed profession

Conformity Ingenious

Curriculum-centred Learner-centred

Interactive Participative

Individualised Community-centred

Delivered wisdom User-generated wisdom

Management Leadership

Public vs private Public with private

Culture as obstacle Culture as capital

AustraliaAustriaCzech RepublicDenmarkFinlandGermanyGreeceHungaryIcelandIrelandItalyJapanNetherlandsNew ZealandNorwayPolandPortugalSlovak RepublicSpainSwedenUnited KingdomUnited States

0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

1995Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)

Tertiary-type A graduation rate

A world of change – higher education

Graduate supply

Cost

per

stu

den

t


0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

1995Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)



United States

Finland

Graduate supply

Cost

per

stu

den

t


0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2000Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)



Australia

FinlandUnited Kingdom


0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2001Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)




0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2002Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)




0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2003Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)




0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2004Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)




0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2005Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)




0 10 20 30 40 50 60 700

5000

10000

15000

20000

25000

30000

2006Ex

pend

iture

per

stu

dent

at t

ertia

ry le

vel (

USD

)



United States

Australia

Finland

1313 E

duca

tion Indic

ato

rs

Pro

gra

mm

e20

09 e

dit

ion o

f Ed

uca

tion a

t a G

lance

Latin America then…

Hanushek 2009

GDP/pop 1960

Years schooling

Asia 1891 4

Sub-Saharan Africa 2304 3.3

MENA 2599 2.7

Latin America 4152 4.7

Europe 7469 7.4

Orig. OECD 11252 9.5

1414 E

duca

tion Indic

ato

rs

Pro

gra

mm

e20

09 e

dit

ion o

f Ed

uca

tion a

t a G

lance GDP/pop

1960Years

schooling

Asia 1891 4

Sub-Saharan Africa 2304 3.3

MENA 2599 2.7

Latin America 4152 4.7

Europe 7469 7.4

Orig. OECD 11252 9.5

Latin America then and now…

GDP/pop 1960

Years schooling

Growth 1960-2000

GDP/pop 2000

Asia 1891 4 4.5 13571

Sub-Saharan Africa 2304 3.3 1.4 3792

MENA 2599 2.7 2.7 8415

Latin America 4152 4.7 1.8 8063

Europe 7469 7.4 2.9 21752

Orig. OECD 11252 9.5 2.1 26147

Hanushek 2009

1515 E

duca

tion Indic

ato

rs

Pro

gra

mm

e20

09 e

dit

ion o

f Ed

uca

tion a

t a G

lance

Latin America then and now…Why quality is the key

Hanushek 2009

GDP/pop 1960

Years schooling

Growth 1960-2000

GDP/pop 2000

PISAscore

Asia 1891 4 4.5 13571 480

Sub-Saharan Africa 2304 3.3 1.4 3792 360

MENA 2599 2.7 2.7 8415 412

Latin America 4152 4.7 1.8 8063 388

Europe 7469 7.4 2.9 21752 492

Orig. OECD 11252 9.5 2.1 26147 500

1616N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Know what you are looking for

The Holy Grail was a well-described object, and there was only one true grail…

1717N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Schooling in the medieval age:

The school of the church

1818N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Schooling in the industrial age:

Uniform learning

1919N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Schooling in the industrial age:

Uniform learning

The challenges today:

Universal quality

Motivated and self-reliant citizens

Risk-taking entrepreneurs, converging and continuously emerging professions tied to globalising contexts and technological advance

2020N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rHow the demand for skills has changed

Economy-wide measures of routine and non-routine task input (US)

1960 1970 1980 1990 200240

45

50

55

60

65 Routine manual

Nonroutine manual

Routine cognitive

Nonroutine analytic

Nonroutine inter-active

(Levy and Murnane)

Mean t

ask

inp

ut

as

perc

en

tile

s of

the 1

960

task

dis

trib

uti

on

The dilemma of assessments:The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource

2323N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r Education today needs to prepare students…

… to deal with more rapid change than ever before…

… for jobs that have not yet been created…… using technologies that have not yet been

invented…… to solve problems that we don’t yet know will

arise It’s about new…

Ways of thinking– involving creativity, critical thinking, problem-solving and

decision-making Ways of working

– including communication and collaboration Tools for working

– including the capacity to recognise and exploit the potential of new technologies

The capacity to live in a multi-faceted world as active and responsible citizens.

2424N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rMathematics in PISA

The real world The mathematical World

A real situation

A model of reality A mathematical model

Mathematical results

Real results

Understanding, structuring and simplifying the situation

Making the problem amenable to mathematical

treatment

Interpreting the mathematical results

Using relevant mathematical tools to solve the problemValidating

the results

2525N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rNational assessment systems differ…

Who is assessed

System assessment

School evaluation

Teacher appraisal

Student assessment

For what? E.g.• Accountabilit

y• Improvement

By whom? • Evaluators• Users of

feedback• Evaluation

agencies

How? Methods and procedures, mix of criteria and instruments• Mapping of

feedback to different units

System

School

Classroom

What?• Inputs• Processes• Outcomes

With whom? • Agents

involved

2626N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rAssessment cultures

Putting the pressure on top of the education system is the easy part, building capacity is harder

Administrative external

Participative/internal

Summative Formative

Interactive, reflective, critical

friend

Survey

InspectorateStandardised assessment

2828N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r…but there are global trends

Multi-layered, coherent assessment systems from classrooms to schools to regional to national to international levels that…

Support improvement of learning at all levels of the education system

Are largely performance-based Make students’ thinking visible and allow for

divergent thinking Are adaptable and responsive to new

developments Add value for teaching and learning by

providing information that can be acted on by students, teachers, and administrators

Are part of a comprehensive and well-aligned continuum, communicate what is expected and hold relevant stakeholders accountable .

• Integrate, synthesize and creatively apply content knowledge in novel situations

• Activate students as owners of their own learning and activate students as learning resources for one another

• Understanding what the assessment reveals about students’ thinking to shape better opportunities for student learning

• Responding to assessments can enhance student learning if tasks are well crafted to incorporate principles of learning

• Capitalise on improved data handling tools and technology connectivity to combine formative and summative assessment interpretations for a more complete picture of student learning

2929N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Know how you will recognise it when you find it

The Alchemists’ stone was to be recognised by transforming ordinary metal into gold…

3030N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Age 19

Age 21

Age 21

048

121620

Level 2Level 3

Level 4Level 5

Increased likelihood of postsec. particip. at age 19/21 associated with PISA reading proficiency at age 15

(Canada)after accounting for school engagement, gender, mother

tongue, place of residence, parental, education and family income (reference group PISA Level 1)

Odds ratioCollege entry

School marks at age 15

PISA performance at age

15

3131N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

20102011201220132014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050205120522053205420552056205720582059206020612062206320642065206620672068206920702071207220732074207520762077207820792080208120822083208420852086208720882089209020912092209320942095209620972098209921002101210221032104210521062107210821092110-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Relationship between test performance and economic outcomes

Annual improved GDP from raising performance by 25 PISA pointsPe

rcent

add

itio

n t

o G

DP

3333N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Implications and lessons learned

The medieval Alchemists’ followed the dictates of a well-established science but that was built on wrong

foundations

The search for the Holy Grail was overburdened by false clues and cryptic symbols

3434N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rFrom assessment-inhibited practice

towards outcome driven reform

Weak focus on processes

Strong focus on processes

Weak outcome-based

management

Strong outcome-based management

Integrated quality management

Good willand trust

External control, uninformed prescription

Deprivation

3535N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rSome criteria used in the world

Coherence Built on a well-structured conceptual base—an

expected learning progression—as the foundation both for large scale and classroom assessments

Consistency and complementarity across administrative levels of the system and across grades

Comprehensiveness Using a range of assessment methods to ensure

adequate measurement of intended constructs and measures of different grain size to serve different decision-making needs

Provide productive feedback, at appropriate levels of detail, to fuel accountability and improvement decisions at multiple levels

Continuity A continuous stream of evidence that tracks the

progress of both individual students .

3737N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rUnderstanding learning

progressions Learning targets

Defining what mastery means for a given skill level Progress variables

Delineate a pathway that characterise the steps that learners typically follow as they become more proficient

Evaluation of students reasoning in terms of the correctness of their solutions as well as in terms of their complexity, validity and precision

Levels of achievement Describing the breadth and depth of the learner’s

understanding of the domain at a particular level of advancement

Learning performances The operational definitions of what student’s

understanding would look like at each of the stages of progress . Wilson, ATC21S

3939N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Context- Personal

- Social/public- Global

Competencies- Identify scientific issues- Explain phenomena scientifically- Use scientific evidence

Knowledge- Knowledge of science- Knowledge about science

Attitudes-Interest in science-Support for scientific enquiry-Responsibility

IdentifyingRecognising issues that can be investigated scientificallyIdentifying keywords in a scientific investigationRecognising the key features of a scientific investigation

ExplainingApplying knowledge of science in a situationDescribing or interpreting phenomena scientifically or predicting change

Using evidenceInterpreting scientific evidence and drawing conclusionsIdentifying the assumptions, evidence and reasoning behind conclusions

OECD Level 6

OECD Level 2 Students can demonstrate

ability to understand and articulate the complex modelling inherent in the design of an investigation.

Students can determine ifscientific measurement can be applied to a given variable in an investigation. Students can appreciate the relationship between a simple model and the phenomenon it is modelling.

Students can draw ona range of abstract scientific knowledge and concepts andthe relationships between these in developing explanations ofprocesses

Students can recall anappropriate, tangible, scientific fact applicable in a simple and straightforward context and can use it to explain or predict an outcome.

Students demonstrateability to compare and differentiate among competing explanations byexamining supporting evidence. They can formulate arguments by synthesising evidence from multiplesources.

Students can point to an obvious feature in a simple table in support of a given statement. They are able to recognise if a set of given characteristics apply to the function of everydayartifacts.

4040N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

rSome methodological challenges

Can we sufficiently distinguish the role of context from that of the underlying cognitive construct ?

Do new types of items that are enabled by computers and networks change the constructs that are being measured ?

Can we drink from the firehose of increasing data streams that arise from new assessment modes ?

Can we utilise new technologies and new ways of thinking of assessments to gain more information from the classroom without overwhelming the classroom with more assessments ?

What is the right mix of crowd wisdom and traditional validity information ?

How can we create assessments that are activators of students’ own learning ?Wilson, ATC21S

4141N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Low policy value

High policy value

Low feasibility High feasibility

Money pits

Must haves

Low-hanging fruits

Quick wins

Examine individual, institutional and systemic

factors associated with performance

Establish the relative standing of students and

schools

Extending the range of competencies through which

quality is assessed

Measuring growth in learning

A real-time assessment environment that bridges the gap between formative and

summative assessment .

Monitor educational progress

Assuming that every new skill domain is orthogonal

to all others

4242P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

See

ing

Japa

nese

sch

ools

thro

ugh

the

pris

m o

f PIS

A

•Principled•Strategic partnership

•Negotiated•Pragmatic .

•Top-down•Antagonistic .

•Leading•Evidence-driven•Achieving high reliability and innovation .

• Enabling• Incentivising

.

•World class performance.

•Continuous learning and innovation .

Good Great

•Accommodating•Evidence-based•Adopting best . practice

•Regulating .•Capacity-building

•Transparency .•Spreading best practice

• Implementing•Accepting evidence•Adopting minimum standards

•Prescribing .• Justifying

• Tackling underperformance

Adequate GoodPoor Adequate

Main focus of assessment

Role of government

Role of professions

Nature of relationship between government

and professions

Phases of development

Main outcomes

• Improvement in outcomes

•Reduction of public anxiety.

•Steady improvement

•Growing public satisfaction .

•Consistent quality•Public engagement and co-production .

Getting the sequencing right

4343N

atio

nal C

onf

ere

nce

on

Nex

t G

ener

atio

n A

sses

smen

tsW

ashi

ngto

n, 8

-9 M

arc

h 2

010

Fu

ture

of

As

sess

men

tsA

ndre

as S

chle

iche

r

Thank you !

www.oecd.org; www.pisa.oecd.org– All national and international publications– The complete micro-level database

email: [email protected] Twitter: @SchleicherEDU

…and remember:

Without data, you are just another person with an opinion

national conference on next generation assessments washington, 8-9 march 2010 future of assessments...

Documents

modern education systems

modern enabling systems

glanceeducation systems

national conference

future of assessmentslessons

national standards

different students

lot of prescription