pisa programme for international student assessment pisa team department of education – ghent...
TRANSCRIPT
PISAProgramme for International Student
Assessment
PISA teamDepartment of Education – Ghent
University – BelgiumBeijing – July 24-25, 2009
http://allserv.ugent.be/~mvalcke/CV/[email protected]
(Partly based on Schleicher, A., 2006)
Structure
• Starting grounds
• Objectives
• Samples – Population
• Quality benchmarks
• Framework
• Typical assessment approach
• National Centres
PISA: starting grounds
• OECD: Organisation for Economical Organisation and Development
• Basis of PISA: original work of OECD related to statistics and indicators about education– late 1980s
– voluntary contributions
– Member engagement through networks
PISA: starting grounds
• Network on educational outcomes– Proposal to study educational outcomes
– Formally started with 11 members in 1996
– Expanded in 1997• New name PISA
• Members bear costs and risks
– Number of countries grows every PISA edition: 2000, 2003, 2006, 2009, …
PISA 2000
OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Liechtenstein, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Spanje, Zweden, Zwitserland, Verenigd Koninkrijk, Verenigde StatenPartnerlanden (niet-OESO): Albanië, Argentinië, Brazilië, Bulgarije, Chili, Hong Kong-China, Indonesië, Israël, Letland, Voormalige Joegoslavische Republiek Macedonië, Peru, Roemenië, Rusland, Thailand
PISA 2003
OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Liechtenstein, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Slovakije, Spanje, Zweden, Zwitserland, Turkije,
Verenigd Koninkrijk, Verenigde StatenPartnerlanden (niet-OESO): Brazilië, Hong Kong-China, Indonesië, Letland, Macao-China, Rusland, Servië &
Montenegro, Thailand, Tunesië, Urugay
PISA 2006
OESO-landen: Australië, Oostenrijk, België, Canada, Tsjechische Republiek, Denemarken, Finland, Frankrijk, Duitsland, Griekenland, Hongarije, Ijsland, Ierland, Italië, Japan, Korea, Luxemburg, Mexico, Nederland, Nieuw-Zeeland, Noorwegen, Polen, Portugal, Slovakije, Spanje, Zweden, Zwitserland, Turkije, Verenigd Koninkrijk, Verenigde StatenPartnerlanden (niet-OESO): Argentinië, Azerbeidzjan, Brazilië, Bulgarije, Chili, Colombia, Kroatië, Estland, Hong Kong-China, Indonesië, Israël, Jordanië, Kyrgyzstan, Letland, Liechtenstein, Litouwen, Macao-China, Qatar, Montenegro, Servië, Roemenië, Rusland, Slovenië, Taipei, Thailand, Tunesië, Uruguay
OECD countries participating from PISA 2000
OECD countries participating from PISA from 2003
OECD partner countries participating from PISA 2000
OECD partner countries participating from PISA 2003
OECD partner countries participating from PISA 2006
PISA country participation
PISA 2000 (32 countries)
PISA 2003 (41 countries)
PISA 2006 (57 countries)
• A three-yearly global educational assessment• What did they learn?• Performance of 15-year-olds• key subject areas, and a range of educational outcomes
• Additionally:• student attitudes to learning, self efficacy beliefs, and
learning strategies • contextual data from students, schools, parents and
systems: policy levers
PISA: Objectives
See further
• Comparing performance within and between countries
• Cross-cultural study
• Central concept: LITERACY• Mathematical literacy• Scientific literacy• Reading literacy
PISA: Objectives
Example: scientific literacy Is defined in terms of an individual’s:• Scientific knowledge and use of that knowledge to…
… identify scientific issues, … explain scientific phenomena, and … draw evidence-based conclusions about science-related issues
• Understanding of the characteristic features of science as a form of human knowledge and enquiry
• Awareness of how science and technology shape our material, intellectual and cultural environments
• Willingness to engage with science-related issues .
PISA:Objectives
• Population of 15 year old pupils
• National samples Representative samples between 3,500 - 50,000 pupils Most federal countries: regional samples
e.g., Flanders versus Wallonia within Belgium PISA covers roughly 90% of the world economy .
PISA: Population versus samples
13
German speaking commmunity
WalloniaFrench speaking commmunity
FlandersDutch speaking commmunity
BelgiumFederal state(10,511.382 inhabitants)
PISA sampling requirements• Population: all 15-year-olds in school
– excludes 15-year-olds out of school– includes 15-year-olds in special education institutions– could exclude up to 5% of 15-year-olds in school
• difficult to reach (e.g. remote schools)• non-participation
– few countries fail to reach required sample in 2003• NZ (5.1%), Denmark (5.3%), UK (5.4%), Canada (6.8%),
Spain (7.3%), US (7.3%)• Sample
– minimum of 150 schools per country– two random samples: schools and replacement schools– if school declines, replacement school is invited– requirements set by countries
• International expertise network building on the participating countries…– Instruments ~ input of > 40 countries (see next presentation)– Cross-national and cross-cultural validity – Analysis of results
• International, natiopnal, regional analyses and reports • Country reviews
• Consortium of research institutionsACER, CITO, ETS, NIER, WESTAT
• Coordinated by OECD and international organisations
PISA: Networks
• Focus on performance in subject areas:– Languages: Reading literacy
• Using, interpreting and reflecting on written material.
– Mathematics: Mathematical literacy• Recognising problems that can be solved mathematically, representing them
mathematically, solving them.
– Sciences: Scientific literacy• Identifying scientific questions, recognising what counts as scientific evidence, using
evidence to draw conclusions about the natural world.
PISA: Objectives
PISA Objectives: cycles
2000 2003 2006 2009 2012
Reading Reading Reading Reading Reading
Maths Maths Maths Maths Maths
Science Science Science Science Science
• Focus on performance in additional domains subject areas:– 2000: Problem Solving
– 2003: ICT literacy
– 2006: Attitudes towards science
– 2009: ERA
PISA: Objectives
PISA key quality benchmarks• Overall performance of education systems
• Equity in the distribution of learning opportunities– Measured by the impact students’ and schools’ socio-economic
background has on performance…
…not merely by the distribution of learning outcomes
• Consistency of performance standards across schools
• Gender differences
• Foundations for lifelong learning
PISA
2000 2003 2006 2009 2012
Reading Reading Reading Reading Reading
Maths Maths Maths Maths Maths
Science Science Science Science Science
Problem solving
ICT literacyAttitudes towards sciences
ERA
PISA framework
Outputs and Outcomes
impact of learning
Quality and distribution of
knowledge & skills
PISA framework
Student background
variables
Mediating variables
Literacy
PISA framework: complex interplay variables
Student background
variables
Mediating variables
Literacy
PISA framework
Social & economic outcomes of
education
Output and performance of
institutions
Quality of instructional delivery
Quality and distribution of
knowledge & skills
Outputs and Outcomes
impact of learning
Instructional settings
LevelB
Schools, other institutions
LevelC
200
300
400
500
600
700
800
-3 -2 -1 0 1 2 3
HighStu
dent
perf
orm
ance
AdvantagePISA Index of social backgroundDisadvantage
Low
PISA framework
Structures, resource allocation
and policies
Social & economic outcomes of
education
The school learning environment
Teaching, learning practices and
classroom climate
Individual attitudes, engagement and
behaviour
Output and performance of
institutions
Quality of instructional delivery
Quality and distribution of
knowledge & skills
Policy Leversshape educational
outcomes
Outputs and Outcomes
impact of learning
Instructional settings
LevelB
Schools, other institutions
LevelC
Domain 2
PISA framework
National education, social & economic
context
Structures, resource allocation
and policies
Social & economic outcomes of
education
Community and school
characteristics
Student learning, teacher working
conditions
Socio-economic background of
learners
Antecedentscontextualise or
constrain ed policy
The school learning environment
Teaching, learning practices and
classroom climate
Individual attitudes, engagement and
behaviour
Output and performance of
institutions
Quality of instructional delivery
Quality and distribution of
knowledge & skills
Policy Leversshape educational
outcomes
Outputs and Outcomes
impact of learning
Instructional settings
LevelB
Schools, other institutions
LevelC
Domain 3Domain 2
Typical PISA assessment
• Information collection– From students
• 3½ hours of main domain assessment
• 1 hour in relation to other subdomain
• 2 hours on paper-and-pencil tasks (subset of all questions)
• ½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation
– From school principals• Questionnaire (school demography, learning environment quality)
• Indirect assssment of classroom variables (teacher, class)BEWARE!! Only adequate if grade based sampling has been applied
PISA National Centre
• Linking with international consortium
• Implementation of framework ~national level
• Reporting to consortium
• Representation during international meetings
PISA International
• Strong prescriptive framework
• Framework, timing, procedures, tools
• National data gathering
• International processing of dataPriority in international release of results
• National (regional) processing of datanext priority level
• Secondary analysis of data: data available
New developments
• Towards electronic assessment
• 2009: first full scale trial– ERA Electronic Reading Assessment
PISAProgramme for International Student
Assessment
PISA teamDepartment of Education – Ghent
University – BelgiumBeijing – July 24-25, 2009
http://allserv.ugent.be/~mvalcke/CV/[email protected]
(Partly based on Schleicher, A., 2006)