use of pisa in quality improvement polices – richard yelland, oecd head of policy advice and...
DESCRIPTION
Presentation at the conference "Quality Education for Better Schools, Results and Future" organized by UNICEF and the Ministry of Education in Podgorica, July 8-10, 2014TRANSCRIPT
OECD EMPLOYER
BRAND
Playbook
1
Programme for
International
Student
Assessment (PISA)
Richard Yelland
2 PISA in brief
• Over half a million students…– representing 28 million 15-year-olds in 65 countries/economies
… took an internationally agreed 2-hour test…– Goes beyond testing whether students can
reproduce what they were taught…
… to assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations
– Mathematics, reading, science, problem-solving, financial literacy
– Total of 390 minutes of assessment material
… and responded to questions on…– their personal background, their schools
and their engagement with learning and school
• Parents, principals and system leaders provided data on…– school policies, practices, resources and institutional factors that
help explain performance differences .
3 PISA in brief
• A shared learning tool for all involved– ‘Crowd sourcing’ and collaboration
• PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies…
… guided by governments on the basis of shared policy interests
– Cross-national relevance and transferability of policy experiences
• Emphasis on validity across cultures, languages and systems
• Frameworks built on well-structured conceptual understandingof academic disciplines and contextual factors
– Triangulation across different stakeholder perspectives
• Systematic integration of insights from students, parents, school principals and system-leaders
– Advanced methods with different grain sizes
• A range of methods to adequately measure constructs with different grain sizes to serve different decision-making needs
• Productive feedback, at appropriate levels of detail, to fuel improvement at every level of the system .
4 The structure of the PISA assessment
2000 2003 2006 2009 2012 2015
Reading Reading Reading Reading Reading Reading
Mathematics
Mathematics
Mathematics
Mathematics
Mathematics Mathematics
Science Science Science Science Science Science
Problem Solving
Digital Reading
Problem Solving, Financial
literacy, Digital Math, Digital
reading
Collaborative Problem Solving,
Financial literacy,
• PISA 2012:
–Student and school questionnaires
• Options:
– ICT questionnaire
–Educational career questionnaire
–Parent questionnaire
5 Questionnaires
6
Climbing Mount Fuji
Mount Fuji is a famous dormant volcano
in Japan.
Mount Fuji is only open to the public for
climbing from 1 July to 27 August each
year. About 200 000 people climb
Mount Fuji during this time.
On average, about how many people
climb Mount Fuji each day?
A. 340
B. 710
C. 3400
D. 7100
E. 7400
PISA 2012 Sample Question 1
7
Percent of 15-year-olds who scored Level 2 or AboveS
hang
hai-C
hina
Sin
gapo
reH
ong
Kon
g-C
hina
Kor
eaE
ston
iaM
acao
-Chi
naJa
pan
Fin
land
Sw
itzer
land
Chi
nese
Tai
pei
Can
ada
Liec
hten
stei
nV
ietn
amP
olan
dN
ethe
rland
sD
enm
ark
Irel
and
Ger
man
yA
ustr
iaB
elgi
umA
ustr
alia
Latv
iaS
love
nia
Cze
ch R
epub
licIc
elan
dU
nite
d K
ingd
omN
orw
ayF
ranc
eN
ew Z
eala
ndO
EC
D a
vera
ge
Spa
inR
ussi
an F
eder
atio
nLu
xem
bour
gIta
lyP
ortu
gal
Uni
ted
Sta
tes
Lith
uani
aS
wed
enS
lova
k R
epub
licH
unga
ryC
roat
iaIs
rael
Gre
ece
Ser
bia
Rom
ania
Tur
key
Cyp
rus*
Bul
garia
Kaz
akhs
tan
Uni
ted
Ara
b E
mira
tes
Tha
iland
Chi
leM
alay
sia
Mex
ico
Uru
guay
Mon
tene
gro
Cos
ta R
ica
Alb
ania
Arg
entin
aB
razi
lT
unis
iaJo
rdan
Qat
arC
olom
bia
Per
uIn
done
sia
0
10
20
30
40
50
60
70
80
90
100
PISA 2012 Sample Question 1
8
Revolving DoorA revolving door includes three wings which rotate within a circular-shaped space. The inside diameter of
this space is 2 metres (200 centimetres). The three door wings divide the space into three equal sectors.
The plan below shows the door wings in three different positions viewed from the top.
The two door openings (the dotted arcs in the diagram) are the same size.
If these openings are too wide the revolving wings cannot provide a sealed
space and air could then flow freely between the entrance and the exit,
causing unwanted heat loss or gain. This is shown in the diagram opposite.
What is the maximum arc length in centimetres (cm) that each door
opening can have, so that air never flows freely between the entrance and
the exit?
Maximum arc length: ____________ cm
PISA 2012 Sample Question 4
9
Percent of 15-year-olds who scored Level 6 or AboveS
hang
hai-C
hina
Sin
gapo
re
Chi
nese
Tai
pei
Hon
g K
ong-
Chi
na
Kor
ea
Japa
n
Mac
ao-C
hina
Liec
hten
stei
n
Sw
itzer
land
Bel
gium
Pol
and
Ger
man
y
New
Zea
land
Net
herla
nds
Can
ada
Aus
tral
ia
Est
onia
Fin
land
Vie
tnam
Slo
veni
a
OE
CD
ave
rag
e
Aus
tria
Cze
ch R
epub
lic
Fra
nce
Slo
vak
Rep
ublic
Uni
ted
Kin
gdom
Luxe
mbo
urg
Icel
and
Uni
ted
Sta
tes
Isra
el
Irel
and
Italy
Hun
gary
Por
tuga
l
Nor
way
Den
mar
k
Cro
atia
Sw
eden
Latv
ia
Rus
sian
Fed
erat
ion
Lith
uani
a
Spa
in
Tur
key
Ser
bia
Bul
garia
Gre
ece
Rom
ania
Uni
ted
Ara
b E
mira
tes
Tha
iland
0
5
10
15
20
25
30
PISA 2012 Sample Question 4
Singapore
Hong Kong-ChinaChinese Taipei
Korea
Macao-ChinaJapan LiechtensteinSwitzerland
NetherlandsEstonia FinlandCanada
PolandBelgiumGermany Viet Nam
Austria AustraliaIrelandSlovenia
DenmarkNew ZealandCzech Republic France
United KingdomIceland
LatviaLuxembourg NorwayPortugal ItalySpain
Russian Fed.Slovak Republic United StatesLithuaniaSwedenHungary
CroatiaIsrael
GreeceSerbiaTurkey
Romania
BulgariaU.A.E.KazakhstanThailand
ChileMalaysia
Mexico410
420
430
440
450
460
470
480
490
500
510
520
530
540
550
560
570
580
Mean score
High mathematics performance
Low mathematics performance
… Shanghai-China performs above this line (613)
Montenegro, with 11 countries performing below
Average performance
of 15-year-olds in
MathematicsFig I.2.13
US
Change in performance between PISA 2003 and 2012
Indonesia
Thailand
Russian Fed.
United States
Latvia
Spain
NorwayLuxembourg
Ireland
Austria
SwitzerlandJapan
Liechtenstein
Korea
Brazil
Tunisia
Mexico
Uruguay
Turkey
Greece
Italy
Portugal
Hungary
Poland
Slovak Republic
OECD average
Germany
Sweden
France
Denmark
Iceland
Czech Republic
New ZealandAustralia
Macao-China
Belgium
Canada
Netherlands
Finland
Hong Kong-China
-4
-3
-2
-1
0
1
2
3
4
5
350 400 450 500 550 600
Ave
rag
e a
nn
ua
l m
ath
em
ati
cs
sc
ore
ch
an
ge
Average mathematics performance in PISA 2003
Montenegro
Imp
rovin
g p
erfo
rma
nc
eD
ete
riora
ting
pe
rform
an
ce
PISA 2003 performance below the OECD averagePISA 2003 performance
above the OECD average
Fig I.2.1811
Mathematics, reading and science Israel, Poland, Portugal, Turkey, Brazil, Dubai
(UAE), Hong Kong-China,
Macao-China, Qatar, Singapore, Tunisia
Mathematics and readingChile, Germany, Mexico, Albania, Montenegro,
Serbia, Shanghai-China
Mathematics and scienceItaly, Kazakhstan, Romania
Reading and scienceJapan, Korea, Latvia, Thailand
Mathematics onlyGreece, Bulgaria, Malaysia,
United Arab Emirates (ex. Dubai)
Reading only Estonia, Hungary, Luxembourg, Switzerland,
Colombia, Indonesia, Liechtenstein, Peru,
Russian Federation, Chinese Taipei
Science onlyIreland
Of the 65 countries 45 improved at least in one subject12
AustraliaAustria
BelgiumCanada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
IcelandIreland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain Sweden
Switzerland
Turkey
UK
US
Singapore
Hong Kong-ChinaChinese Taipei
Macao-China
Liechtenstein
Viet Nam
Latvia
Russian Fed.Lithuania
Croatia
SerbiaRomania
Bulgaria United Arab Emirates
Kazakhstan
Thailand
Malaysia
Montenegro
02468101214161820222426
2012Shanghai-China
Socially equitable
distribution of learning
opportunities
Strong socio-economic
impact on student
performance
Performance and equity:
a tradeoff ?
0
10
20
30
40
50
60
70
80
Sh
an
gh
ai-
Chin
aH
on
g K
on
g-C
hin
aM
acao
-Chin
aV
iet
Nam
Sin
ga
pore
Ko
rea
Chin
ese
Ta
ipe
iJap
an
Lie
chte
nste
inS
witze
rlan
dE
sto
nia
Ne
the
rlan
ds
Po
lan
dC
an
ad
aF
inla
nd
Be
lgiu
mP
ort
ug
al
Ge
rma
ny
Tu
rke
yO
EC
D a
vera
ge
Italy
Sp
ain
Latv
iaIr
ela
nd
Au
str
alia
Th
aila
nd
Au
str
iaL
uxe
mb
ourg
Czech
Rep
ublic
Slo
ve
nia
Un
ite
d K
ing
do
mL
ith
uan
iaF
rance
Norw
ay
Ice
lan
dN
ew
Zea
land
Ru
ssia
n F
ed
.U
nite
d S
tate
sC
roa
tia
Den
ma
rkS
wed
en
Hun
ga
ryS
lova
k R
ep
ub
licM
exic
oS
erb
iaG
ree
ce
Isra
el
Tu
nis
iaR
om
an
iaM
ala
ysia
Indo
nesia
Bu
lga
ria
Ka
za
kh
sta
nU
rug
ua
yB
razil
Costa
Ric
aC
hile
Colo
mbia
Mo
nte
ne
gro
U.A
.E.
Arg
en
tina
Jord
an
Pe
ruQ
ata
r
%
Percentage of resilient students
More than 40
% resilient Between 20%-40% of resilient students Less than 20%
Fig II.2.414
Socio-economically disadvantaged students not only score lower in mathematics, they also report lower levels of engagement, drive, motivation and self-beliefs. Resilient students break this link and share many characteristics of advantaged high-achievers.
A resilient student is situated in the bottom quarter of
the PISA index of economic, social and cultural
status (ESCS) in the country of assessment and
performs in the top quarter of students among all
countries, after accounting for socio-economic status.
-50
-40
-30
-20
-10
0
10
20
30
Jo
rdan
Qata
rT
haila
nd
Ma
laysia
Icela
nd
U.A
.E.
Latv
iaS
inga
po
reF
inla
nd
Sw
ede
nB
ulg
aria
Ru
ssia
n F
ed.
Alb
an
iaM
on
ten
eg
roL
ithu
ania
Kaza
kh
sta
nN
orw
ay
Ma
ca
o-C
hin
aS
loven
iaR
om
ania
Pola
nd
Ind
one
sia
Un
ited
Sta
tes
Esto
nia
Ch
inese
Taip
ei
Sha
ngh
ai-
Ch
ina
Belg
ium
Tu
rke
yG
reece
Fra
nce
Hung
ary
Serb
iaS
lovak R
epu
blic
Vie
tna
mC
ana
da
Ne
the
rla
nds
OE
CD
ave
rag
eP
ort
ug
al
Uru
gua
yC
roa
tia
Isra
el
Cze
ch R
ep
ub
licA
ustr
alia
Un
ited
Kin
gd
om
Sw
itze
rla
nd
Germ
any
Arg
en
tin
aD
enm
ark
Me
xic
oN
ew
Ze
ala
nd
Tu
nis
iaIr
ela
nd
Ho
ng
Kon
g-C
hin
aS
pa
inB
razil
Ja
pa
nK
ore
aIt
aly
Peru
Austr
iaL
iech
tenste
inC
osta
Ric
aC
hile
Luxe
mb
ou
rgC
olo
mb
ia
Sc
ore
-po
int
dif
fere
nc
e (
bo
ys
-gir
ls)
Gender differences in mathematics performance Fig I.2.25
Boys perform better than girls
Girls perform better than boys
15
Resources make a difference…
…but only up to a point
16
Spending per student from the age of 6 to 15 and
mathematics performance in PISA 2012
Slovak Republic
Czech RepublicEstonia
Israel
Poland
Korea
Portugal
New Zealand
CanadaGermany
Spain
France
Italy
Singapore
Finland
Japan
Slovenia Ireland
Iceland
Netherlands
Sweden
Belgium
UK
AustraliaDenmark
United States
Austria
Norway
Switzerland
Luxembourg
Viet Nam
Jordan
Peru
Thailand
Malaysia
Uruguay
Turkey
Colombia
Tunisia
MexicoMontenegro
Brazil
Bulgaria
Chile
CroatiaLithuania
Latvia
Hungary
Shanghai-China
R² = 0.01
R² = 0.37
300
350
400
450
500
550
600
650
0 20 000 40 000 60 000 80 000 100 000 120 000 140 000 160 000 180 000 200 000
Ma
the
ma
tic
s p
erf
orm
an
ce
(sc
ore
po
ints
)
Average spending per student from the age of 6 to 15 (USD, PPPs)
Cumulative expenditure per student less than USD 50 000
Cumulative expenditure per student USD 50 000 or more
Fig IV.1.817
Hong Kong-China
Brazil
Uruguay
Croatia
Latvia
Chinese Taipei
Thailand
Bulgaria
Jordan
Macao-China
UAE
Argentina
Indonesia
Kazakhstan
Peru
Costa RicaMontenegro
Tunisia
Qatar
Singapore
Colombia
MalaysiaSerbia
Romania
Viet Nam
Shanghai-China
USA
Poland
New Zealand
Greece
UK
Estonia
Finland
Slovak Rep.
Luxembourg
Germany
AustriaFrance
Japan
TurkeySweden Hungary
AustraliaIsrael
Canada
Ireland
Chile
Belgium
SpainDenmark
Switzerland
Iceland
Slovenia
PortugalNorway
Mexico
Korea
Italy
R² = 0.19
300
350
400
450
500
550
600
650
700
-0.500.511.5
Ma
the
ma
tic
s p
erf
orm
an
ce
(sc
ore
po
ints
)
Equity in resource allocation(index points)
Countries with better performance in mathematics tend
to allocate educational resources more equitably
Greater
equityLess
equity
Adjusted by per capita GDP
Fig IV.1.11
30% of the variation in math performance across OECD countries is explained by the degree of similarity of
educational resources between advantaged and disadvantaged schools
OECD countries tend to allocate at least an equal, if not a larger, number of teachers per student to disadvantaged schools; but disadvantaged schools tend to have great difficulty in attracting qualified teachers.
Video series on
Strong Performers and
Successful Reformers in
Education
http://www.pearsonfoundation.org/oecd
What’s next?
PISA 2015
20
• Main subject: Science
• Number of participants : 72
• Field trials in 2014
• Main survey 2015
• Results released in December 2016
PISA 201521
• Engagement of all is important:
– Policy-makers
– Teachers and Schools
– Students and Parents
– Media
– Research community
22
Thank you !
Find out more about PISA at www.pisa.oecd.org
• National and international publications
• The complete micro-level database
With acknowledgements to the PISA team
Email: [email protected]