For Official Use EDU/PISA/GB(2012)28 Organisation de Coopération et de Développement Économiques Organisation for Economic Co-operation and Development 12-Oct-2012

___________________________________________________________________________________________

_____________ English - Or. English DIRECTORATE FOR EDUCATION

PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT

Governing Board

PROGRESS REPORT ON QUESTIONNAIRE DEVELOPMENT AND DESIGN

34th meeting of the PISA Governing Board

29-31 October 2012

Madrid, Spain

Andreas Schleicher; E-mail: andreas.SCHLEICHER@oecd.org; Tel: +33.1.45.24.93.66

JT03328578

Complete document available on OLIS in its original format

This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of

international frontiers and boundaries and to the name of any territory, city or area.

ED

U/P

ISA

/GB

(20

12

)28

Fo

r Officia

l Use

En

glish

- Or. E

ng

lish

EDU/PISA/GB(2012)28

2

TABLE OF CONTENTS

1. Conceptual background ........................................................................................................................ 3 1.1 The PISA approach to large scale assessment .................................................................................... 4 1.2 Overarching conceptual framework for PISA 2015 ........................................................................... 5 1.3 Determining priorities for context assessment in PISA 2015............................................................. 8

2. Questionnaire Development from April to September, 2012 ............................................................. 10 Module 1: Teacher qualification and professional knowledge, and the new Teacher Questionnaire .... 11 Module 2: Science Teaching Practices ................................................................................................... 12 Module 3: School-level learning environments for science ................................................................... 12 Module 4: Science-related outcomes ..................................................................................................... 13 Module 7: Student SES, family and home background ......................................................................... 13 Module 9: Educational pathways in early childhood, Module 14: Parental involvement, and the Parent

Questionnaire option .............................................................................................................................. 13 Module 10: Domain-general student behaviour and attitudes ................................................................ 14 Module 12: Learning Time and Curriculum, and the Educational Career optional questionnaire ......... 15 Module 15: School leadership and management .................................................................................... 16 Module 19: Assessment, evaluation, and accountability....................................................................... 16

3. Questionnaire design and rotation in the Main Study StQ ................................................................. 17 3.1 Designing the PISA 2015 Field Trial ............................................................................................... 17 3.2 Time restrictions and design implications for the Main Study ......................................................... 18

REFERENCES .............................................................................................................................................. 20

APPENDIX: DRAFT OVERVIEW OF FIELD TRIAL CONTENT .......................................................... 22

Tables

Table 1. Questionnaire content used in .................................................................................................. 5 Table 2. Distribution of content ............................................................................................................. 9

Figures

Figure 1. Modular structure of the PISA 2015 context assessment design ............................................. 7

EDU/PISA/GB(2012)28

3

PROGRESS REPORT ON QUESTIONNAIRE DEVELOPMENT AND DESIGN

Introduction

1. This document provides an update on work on the PISA 2015 context questionnaire framework

and questionnaire content, following feedback from the previous two PGB meetings and from further

discussions in the Questionnaire Expert Group. The document outlines the content for the 2015 core and

optional contextual questionnaires and discusses issues relating to design and sampling. In particular, the

document:

Describes the policy-driven overarching conceptual framework that drives questionnaire

development in PISA 2015 and outlines criteria for choosing the questions to be included in the

field trial and main study context questionnaires.

Outlines progress in questionnaire development for each of the policy modules and discusses

design implications (e.g. mode or sampling) for each of the core and optional questionnaires.

Closes with a discussion of the implications of design issues (e.g. timing and rotation of the

student questionnaire in the main study).

Lists (in an appendix) all of the constructs under consideration, by module and questionnaire

instrument.

2. The PGB is invited to:

REVIEW the draft frameworks and content for the PISA 2015 context questionnaires;

PROVIDE DIRECTIONS for the further development of the PISA 2015 core and optional

contextual questionnaires.

3. In particular, the PGB is asked to advise on:

The criteria set out in paragraph 24 for content reduction.

The recommendation in paragraph 73 to have one version of the Student Questionnaire in the

Main Survey, with no rotated content.

1. Conceptual background

4. The questionnaire design for PISA 2015 builds on the frameworks set up in earlier waves and

intends to serve policy needs by providing two types of knowledge: (1) indicators used to evaluate and

monitor educational systems, and (2) multi-level (student, school, and system) models that analyse

relationships between different components of the educational system, especially between context factors

and student outcomes.

5. The main innovation in the development of the PISA 2015 Questionnaire Framework and the

questionnaire instruments is a much closer relationship with policy interests. This was achieved by

focusing the development of new material on a series of modules addressing major policy areas, after

consultation with the PGB to determine their priorities in the development of this new material. As a result,

EDU/PISA/GB(2012)28

4

questions on teaching and learning, including learning time allocation and assessment practices, will

receive more attention, as well as school policies (evaluation, leadership, school management, and school

curriculum), and non-cognitive outcomes (both general and science-related). At the same time, it was

proposed that all policy areas should receive as much coverage as possible. Also, comparing new PISA

results against previous cycles becomes increasingly important, as PISA is maturing as a data base for

long-term educational monitoring and trend analysis.

6. As a result, PISA 2015 will address a very broad range of topics - a mixture of new and well

established content -, some of them being presented in more detail than others. To cope with the amount of

material, the Main Survey design for the Student Questionnaire will need to be revisited, possibly adjusting

the administration time, once exact time estimates are available from the Field Trial.

7. Computer-based delivery will allow for new, interactive item formats and questions being

tailored to the respondent. These innovations will support enhanced monitoring of policy-relevant

indicators, and more sophisticated analyses of patterns and relationships both within and between

countries. Computer-based delivery will also enable research on the processes of filling out the

questionnaire, helping to enhance the validity of the PISA context assessment.

1.1 The PISA approach to large scale assessment

8. In the first assessment cycle, PISA mainly focused on the installation of procedures and the step-

by-step implementation of the three major domains of assessment: Reading in 2000, Mathematics in 2003,

and Science in 2006. Assessment frameworks had to be developed, negotiated with stakeholders, and

implemented. PISA has also been recognised for its detailed, theory-based assessment of family

background, socio-economic status, and immigration background, allowing for sophisticated analyses of

equity issues within and across countries. A lot of effort went into the definition and operationalisation of

individual student background indicators, finally leading to the establishment of a powerful, integrated

index of economic, social, and cultural status for each student (ESCS).

9. Furthermore, PISA has covered specific aspects of school environments and out-of-school

learning opportunities. With regard to science education – which will be the major domain of student

assessment in PISA 2015 - PISA 2006 covered science-related learning opportunities, teaching practices,

and school-level context. In addition to these measures of educational processes, PISA 2006 captured

various kinds of non-cognitive learning outcomes such as student motivation, student engagement, beliefs,

and aspirations. Thus, with PISA 2006, the so-called “Background Questionnaires” had become a mixture

of (a) general student background variables and (b) scales covering a variety of domain-specific processes

and outcomes. At the same time, analytical techniques and reporting improved substantially. The initial

report on PISA 2006, “Science Competencies for Tomorrow's World” (OECD, 2007), was a breakthrough

in that it established multi-level modelling estimates of school effects explaining student outcomes and the

degree of equity, both within and across countries.

10. The second wave of PISA cycles, starting with PISA 2009, established a coherent overarching

framework for context questionnaire development. The PISA 2009 Questionnaire Framework (OECD,

2009) was built on a strong and broad knowledge base grounded in educational effectiveness research. The

PISA 2009 report, organised into six volumes, was based on even more diverse and more sophisticated

analyses (see Table 1). In addition to student achievement, non-cognitive outcomes such as student

engagement were studied in detail, together with cognitive and meta-cognitive strategies. Moreover, the

impact of background variables, individual support factors, school level input, processes and policies, as

well as system level factors was reported – all assessed by student and school questionnaires. Enhanced

reporting on trends both for student achievement and for context measures constituted another important

improvement in PISA 2009.

EDU/PISA/GB(2012)28

5

Table 1. Questionnaire content used in "PISA 2009 Results: 'What Students Know and Can Do'". (OECD 2010)

Volume I: Student Performance in Reading, Mathematics and Science: Student background: Gender Volume II: Overcoming Social Background: Equity in Learning Opportunities and Outcomes. Student background: Socio-economic background, gender, immigration status (language spoken at home, age of arrival, country of origin) Individual support: Parental support (at beginning of primary education/at age 15), pre-primary education (attendance, quality) Volume III: Learning to Learn: Student engagement, strategies and practice Student background: Socio-economic background, gender, immigration status, language spoken at home Outcomes: Enjoyment of reading, time and material used for reading, metacognition (awareness of strategies), self-reported use of reading strategies (memorisation, elaboration, control) Volume IV: What makes a school successful? Resources, policies and practices Student background: Socio-economic background, age of school entry, grade repetition Student-reported processes: learning time (previous education, learning time at school, enrichment/remedial education, after-school lessons), teacher-student relations, disciplinary climate, teacher’s stimulation of reading engagement School input, policies and processes (reported by the principal): type of school (public/private), number of programmes, class size, educational resources (e.g., ICT, library), school responsibility for assessment and

curriculum/for resource allocation, extra-curricular activities provided, school admittance/grouping/transfer policies, assessment practices/purposes, use of achievement data, school accountability, methods for monitoring teachers, teacher and student behaviour, parent involvement and expectations, leadership.

11. The Questionnaire Expert Group (QEG) for PISA 2012 set out to further develop and clarify the

goals, the policy needs, the opportunities as well as the challenges of PISA. The PISA 2012 Questionnaire

Framework (EDU/PISA/GB(2010)23) acknowledged that PISA, as a policy-oriented programme, has to

serve the needs of policy makers in participating countries.

1.2 Overarching conceptual framework for PISA 2015

12. The Core 6 proposal for Questionnaire Development in PISA 2015 identified four broad areas of

interest: i) Outcomes, ii) Student background, iii) Teaching and learning and, iv) School policies and

governance.

Outcomes

13. Educating a person means fostering his or her individual development as a unique, self-

determined, knowledgeable person who gradually gains in ability to participate in society. As each PISA

assessment is a cross-sectional study, PISA cannot capture developmental processes, but PISA serves as a

snapshot of development at the age of 15 years. This snapshot of course includes an assessment of literacy

and life skills, but in addition to these cognitive outcomes, factors such as motivation, self-regulation,

attitudes, and beliefs are important educational outcomes. Recent scientific research on education has

highlighted that non-cognitive factors present important skills for individual and social development

(Heckman, Stixrud and Urzua, 2006; Almlund, Duckworth, Heckman and Kautz, 2011). Therefore, PISA

questionnaires also address non-cognitive outcomes. In PISA 2015, with collaborative problem solving

being one of the cognitive domains, dispositions that are relevant for collaboration should be part of the

design.

EDU/PISA/GB(2012)28

6

Student background

14. In addition to the snapshot of cognitive and non-cognitive outcomes, PISA gathers retrospective

and prospective information about educational pathways and careers across the lifespan. In recent years,

researchers as well as public debates in many countries have stressed the importance of early childhood

education (Barnett, 2008; Blau and Curie, 2006; Cunha, Heckman, Lochner and Masterov, 2006; Sammons

et al. 2011). Therefore, PISA aims at collecting retrospective information on early childhood education.

However, validity of such data gathered from 15-year old students is limited; therefore, the optional Parent

Questionnaire will be used for this purpose. In order to understand educational careers, family background

such as the socio-economic status and immigrant background should be taken into account. The

distribution of educational opportunities and outcomes can help describe the extent to which countries

succeed in granting equal opportunities to their students.

Teaching and Learning

15. School-based instruction is the core process of formal, systematic education. Therefore, policy

makers need information on teaching, learning, and organisation of schools. To increase explanatory power

of the study, assessment of teaching and learning tends to focus on the major domain of assessment, i.e.

science in PISA 2015. The knowledge base of school effectiveness research (Scheerens and Bosker, 1997;

Creemers and Kyriakides, 2008) allows for the identification of core factors, i.e. teachers’ qualifications

and professional activities, teaching practices, and generally learning opportunities provided both within

and outside of school. It is a challenge to address teacher and teaching-related factors in PISA, but student

and school questionnaires may provide at least some proximal indicators, and the optional Teacher

Questionnaire will significantly enhance the scope of the design. Both science-related and general

information on teaching are indispensable if PISA aims at covering educational factors that are strongly

related to student achievement.

School policies and governance

16. As policy makers have a limited direct influence on teaching and learning processes, information

on school-level factors which help improve schools, and thus indirectly improve student learning, shall

have high priority. As to teacher and teaching variables, school effectiveness research has built a strong

knowledge base showing that certain “essential supports” (Bryk, Sebring, Allensworth, Luppescu and

Easton, 2010) drive school improvement. These essentials are professional capacity, a well-organised

curriculum, parental and community involvement, school climate and leadership. All of these will be

addressed within the PISA 2015 questionnaire design. In addition, resources (budget, personnel) and their

distribution between schools are important in educational policy making.

17. To meet policy needs directly, PISA also needs to address issues related to governance on the

system level (Hanushek and Woeßmann 2011; Woeßmann, Lüdemann, Schütz and West, 2007). “Locus of

decision making” measures and accountability practices describe core aspects of governance, namely the

distribution of power and control between central and local institutions. Allocation, selection, choice, as

well as assessment and evaluation are the basic processes deployed by policy makers and/or school

administrators to steer school quality and school improvement.

18. These four areas were differentiated into 19 more fine-grained policy issues, which were

accepted by the PISA Governing Board (at its meeting in October 2011) as the building blocks of the PISA

2015 design for context assessment. Figure 1 provides a schematic overview of this modular structure. It

also shows (circles) how PISA covers all levels of activities related to education: the individual and

classroom level (teaching and learning), the school level, and the system (governance) level.

EDU/PISA/GB(2012)28

7

Figure 1. Modular structure of the PISA 2015 context assessment design

19. Because PISA intends to support educational policy making, the framework can be compared

with one of the most comprehensive pieces of recent international research on education policy. Sykes,

Schneider and Plank (2009) provide 76 chapters with state-of-the-art reviews of educational policy

research. The section on “policy implications of educational resources, management, and organisation”

mentions eight policy issues, e.g. class size, equity policies, school choice, and resources. All of these are

covered by the modules provided by this framework. The section on “teaching and learning policy” covers

10 issues, such as opportunity-to-learn and teacher quality, which are also covered by the modules – with

the exception of demography. The sections on “educational access and differentiation” as well as “politics

and the policy process” each include 4 or 5 issues covered in our modules, such as early childhood

education, shadow (out-of-school) education, assessment policies and ethnicity. Still, a few differences can

be found between the context assessment framework and the Handbook of Educational Policy Research.

PISA does not relate to research on policy processes and specific actors/stakeholders (e.g., the role of

teacher unions). The assumption followed in this framework is that this is so because PISA is not meant to

provide data on decision-making processes. On the other hand, the Handbook of Educational Policy

Research does not include any chapter on non-cognitive outcomes, which nevertheless are important

reporting categories in PISA.

20. To sum up, the modular approach to context assessment in PISA 2015 allows for broad coverage

of policy issues and related research questions. However, there are strict limits to length of questionnaires

EDU/PISA/GB(2012)28

8

and the time needed to fill them out. Therefore, in addition to breadth, PISA needs to focus on the most

important issues (modules, or constructs within modules).

1.3 Determining priorities for context assessment in PISA 2015

21. The PISA Governing Board was asked to indicate the top six priority modules for further

developmental work, based on policy relevance and the need for improvement from previous cycles. In one

way or another, all topics will be covered in the PISA 2015 questionnaires, but more emphasis will be

devoted to those priority areas identified by the PGB.

22. The following six areas received the most votes for high policy relevance and need of further

development work for PISA 2015 in order of popularity:

19. Assessment, evaluation and accountability

02. Science teaching practices

04. Science-related outcomes: motivation, attitudes, beliefs, strategies

12. Learning time and curriculum

10. Domain-general student behaviour and attitudes

15. Leadership and school management

In addition to these priorities, four modules are assigned medium priority:

01. Teacher qualification and professional knowledge

09. Educational pathways in early childhood

03. School-level learning environments for science

07. Student SES, family and home background

The following modules were rated of lowest priority. While these will be included to some extent in

the questionnaires, there has been less development in these areas:

13. School climate: Interpersonal relations, trust, expectations

08. Ethnicity and migration

11. Student dispositions related to collaborative problem solving

05. Out-of-school science experience

06. Science career

16. Resources

14. Parental involvement

18. Allocation, Selection and choice

17. Locus of decision making within the school system

23. Independently, the PISA 2015 Questionnaire Expert Group (QEG) had delivered a similar rating.

Taking into account the results of the PGB rating exercise as well as suggestions from the QEG,

EDU/PISA/GB(2012)28

9

questionnaire development focused on the high priority and (to a lesser extent) the medium priority

modules.

24. Besides strengthening the alignment with policy needs, a clear focus on high and medium priority

modules helps limit the length of the questionnaires. Another method of keeping the size of material

manageable is distributing it across different questionnaires, such as Student and School Questionnaire,

optional Parent and Teacher Questionnaires, and short, optional add-ons to the Student Questionnaires

(Educational career, ICT familiarity), as shown in table 2. As a rule, information will be gathered from

those stakeholders who can provide the most valid perspective.

Table 2. Distribution of content

Module StQ ScQ PQ TQ ICT EC

1 Teacher education and professional knowledge

2 Teaching practices

3 School environment for science learning

4 Science-related non-cognitive outcomes

5 Out-of-school science learning experiences

6 Science career

7 Student SES, family and home background

8 Ethnicity and migration

9 Educational pathways in early childhood education

10 General non-cognitive outcomes

11 Dispositions to CPS

12 Learning time and curriculum

13 School climate

14 Parental involvement

15 Leadership and school management

16 Resources

17 Locus of decision making

18 Admission, selection and choice

19 Assessment, evaluation, and accountability

25. As a third step in content reduction, constructs are prioritised according to the following

principles:

EDU/PISA/GB(2012)28

10

In as far as possible, questions and items with proven validity and measurement quality

(established measures) will be retained from previous PISA cycles, especially from PISA 2006,

to allow for trend reporting.

Questions that serve the needs of multiple modules are preferred for reasons of efficiency.

Short questions will be preferred to longer questions.

Questions that can be expected to work across cultures, and/or may help to describe

country/culture differences, will receive higher priority.

If quality is otherwise equal, constructs will be retained that are more strongly related to student

outcomes.

If there are two or more similar constructs, all but one will be dropped.

26. A reduced, but still relatively large set of questions will be submitted to the PISA 2015 Field

Trial which will take place in 2014. After the Field Trial, a final reduction will be conducted (again in

accordance with above described procedures).

2. Questionnaire Development from April to September, 2012

27. Two meetings of the Questionnaire Expert Group (QEG) took place in 2012, one in May 2012

and one in August 2012.

28. The May meeting focused on the research and policy background of high and medium priority

modules, discussing both conceptual and methodological innovations. In preparation for that meeting, The

PISA consortium (Core 6) staff contracted questionnaire developers and provided background papers for

10 modules and four international options. These papers reviewed relevant research and experience from

previous PISA cycles and laid out conceptual foundations for further development.

29. Based on recommendations from the May QEG meeting, PISA consortium (Core 6) staff and

contracted developers started to revise existing PISA measures and develop new questionnaire material,

once again with a clear focus on high priority modules. At this stage of work, no limitations regarding

breadth and depth were applied, because the goal was to procure a wide set of measures – sometimes even

including several alternative approaches to cover a specific construct. The QEG was thus to receive a wide

array of possible high-quality questions and items, which could then be discussed, reviewed, prioritised,

and – where needed - selected by the QEG. 16 working papers were submitted to the QEG meeting in

August, 2012, comprising some 368 measures across all 19 modules and the international questionnaire

options. At their August meeting, the QEG made a first selection. Following their advice and comments

received from the Secretariat, PISA consortium (Core 6) staff are currently engaged in a second wave of

questionnaire development, revising the proposed material and in some cases also developing new material

to meet requests from the QEG and/or the Secretariat. Results from this second wave of questionnaire

development will be discussed by the QEG in December, 2012 during a series of webinars. In January

2013, PISA consortium (Core 6) staff will open a web-based forum for NPMs, asking for overall

comments and discussion of specific issues. This forum will provide an opportunity for countries to

provide comments on cultural appropriateness and national feasibility of new measures. PGB members and

the Secretariat will receive the final instruments in March 2013 for discussion at the April, 2013, meeting

of the PGB, and for implementation, translation, and adaptation for the Field Trial.

EDU/PISA/GB(2012)28

11

30. To provide a more detailed insight into this on-going process, the following section describes

conceptual foundations and questionnaire development for the high and medium priority modules. Also,

the innovations planned within related international options will be explained in this context.

31. A full overview of material currently discussed within the PISA consortium is presented in the

appendix.

Module 1: Teacher qualification and professional knowledge, and the new Teacher Questionnaire

32. Teachers are at the core of school instruction. Many studies have demonstrated a clear influence

of teaching practices and other teacher-related factors on student learning and outcomes. In recent years,

this relation has received growing interest. Many stakeholders are valuing the central role teachers play in

education, also regarding their role as school leaders and key sources of innovation. The PISA 2015

Questionnaire Framework particularly pursues this issue in two Modules: While Module 2, “Teaching

practices”, deals with professional behaviour within the classroom, Module 1, “Teacher qualification and

professional knowledge”, deals with background variables, individual qualification and competence, as

well as professional practices – such as collaboration and professional development - on the school level.

33. This layout of teacher-related factors in the PISA 2015 Questionnaire Framework is in line with

OECD’s Teaching and Learning International Survey (TALIS), which distinguishes between the teachers’

tasks in terms of classroom practices on the one hand and teachers’ professional activities on the other

hand.

34. In PISA, teacher-related information so far has been gathered from school administrators and

students only. The School Questionnaire asked about initial education of recruited staff and participation in

professional development. These measures will be addressed in PISA 2015 as well, and information on

policies regarding both issues will be added. Also, a scale on students’ perception of teacher engagement

has been adapted from the research literature. Thus, there will be some coverage of Module 1 issues in the

standard questionnaires. In addition, PISA 2015 will include an optional Teacher Questionnaire (TQ) to

tackle this high priority policy module in more detail.

35. The implementation of Module 1 constructs will become much more feasible with the

introduction of this International Option. Teachers themselves will provide much more valid information

on their background, initial education, professional career and professional development activities than

school administrators could do. It will thus be possible to address policy issues of high relevance:

recruitment, selection, and allocation of teachers to schools; career changers; teachers teaching science

classes without any formal training for science education; fluctuation among the teaching staff.

36. An important feature of the Teacher Questionnaire (TQ) in PISA will be the split of the sample:

Up to 10 science teachers will be sampled (TQ-Science), as well as 10 or more teachers teaching 15-year

old students in other subjects (TQ-General). Both groups will receive specific questionnaires, with an

overlap for background variables (TQ-Core). Thus, science teachers can be asked about science-specific

aspects, such as the relative coverage of content knowledge, pedagogical content knowledge (i.e.

knowledge about student learning, typical misconceptions etc.) and general pedagogical knowledge (e.g.,

classroom management techniques) – information that will be gathered through an interactive question

which allows respondents manipulate a pie chart to provide this information. Also, questions about initial

education and professional development will be tailored to science teachers in one case, and otherwise

address more general issues.

37. As discussed at the PGB when the TQ option was submitted in April 2012, and at the QEG, it

will not be possible to measure teachers’ professional knowledge in PISA. However, related beliefs,

EDU/PISA/GB(2012)28

12

especially self-efficacy beliefs, and non-cognitive factors, such as teacher enthusiasm and job satisfaction,

will be addressed. To check the validity of teachers’ self-reports of enthusiasm, (a) anchoring vignettes will

be provided, and (b) teacher engagement/enthusiasm will also be covered from the student perspective.

38. Finally, teachers will provide information on other policy issues, such as leadership, school

climate, parental involvement, and assessment practices.

Module 2: Science Teaching Practices

39. In line with the PISA approach of measuring scientific literacy as the capacity of students to

identify scientific questions, to explain phenomena scientifically, to use scientific evidence, and to value

scientific arguments, most science education experts nowadays recommend so-called inquiry-based

teaching as the via regia for fostering science literacy. Accordingly, PISA 2006 focused on various aspects

of inquiry-based teaching when asking students about their classroom learning experience. Four scales

were established: (1) interactive science teaching, (2) hands-on activities, (3) student investigations, and

(4) real-life applications. However, re-analyses showed that these scales were highly correlated. Also, the

QEG recommended extending the coverage of science teaching practices to capture learning environments

that are not aligned with the inquiry paradigm.

40. As a first step, PISA consortium (Core 6) experts revised the scales, shortening them and

aligning them with more recent research that identifies four aspects of inquiry-based teaching: procedural

aspects (e.g., students designing their own experiments), social aspects (e.g., class debating alternative

hypotheses), epistemic aspects (e.g., drawing conclusions from observations), and real life applications.

41. Secondly, two scales were added that ask about other kinds of science teaching practices. One

scale covers direct instruction (e.g., teacher providing lectures and designing well-structured practice

sessions) which is very popular, and has been shown to have positive effects especially on low-achieving

students. Another scale, named “opportunity to learn”, covers various aspects of media use, instructional

methods (such as teacher vs. student presentations) and interactive settings (e.g., small group vs. whole

class discussion).

42. All the measures discussed above are rather close to observable behaviour, mostly teacher

practices. Research on the quality of teaching, however, suggests that “latent” dimensions of teaching

quality are most predictive for student outcomes. Following experience with mathematics teaching quality

in PISA 2012, measures of teacher support, disciplinary climate, and cognitive activation (i.e. challenging

tasks) in science classes have been developed. Classroom discipline and teacher support have been

measured in most PISA cycles except for 2006, and have been shown to be highly correlated to student

achievement and student motivation, respectively. Both will be adapted to science education in the PISA

2015 Field Trial.

43. Student perceptions proved to be valid indicators of teaching practices and teaching quality in the

research literature. However, with an optional Teacher Questionnaire being introduced, there will also be a

scale reporting on classroom practices from teachers’ point of view. Teachers will be asked how they

perceive “inquiry-based science education”, and to what extent they implement its components. Teachers’

perceptions will be addressed by a set of vignettes, which can also serve as anchoring vignettes for

teachers’ self-reports, allowing to correct for response bias.

Module 3: School-level learning environments for science

44. Conceptually, this module has quite some overlap with other modules dealing with school-level

factors, such as Module 12, Learning Time and Curriculum, Module 15, School Leadership and

Management, or Module 19, Assessment, Evaluation, and Accountability (see below for brief reports on

EDU/PISA/GB(2012)28

13

those). In addition to those, the QEG requested some questions in the School Questionnaire that would

directly focus on the status of science education in the school, and available resources. Accordingly, a

question has been developed to investigate the overall value of science within the school (Is science

education a shared priority for stakeholders?), along with questions on resources available for science

education: size of teaching staff per science area; physical resources such as laboratories and equipment for

student experiments; and cooperation with science museums, companies and other external partners.

Module 4: Science-related outcomes

45. Positive attitudes towards science and technology, but also motivation, value beliefs and

strategies that motivate students to engage in scientific issues are important educational and policy goals in

many countries. Moreover, such attitudes affect students’ learning and performance as well as their choice

of courses and future careers. However, the relationship between attitudes, motivation, value beliefs and

strategies on one hand and science performance on the other hand is reciprocal. They can cause higher

science performance as well as being a consequence of higher science performance. Therefore, they are

important outcomes of education themselves.

46. PISA 2015 will not use the “embedded measures” of content or task-specific interest that were

used in some PISA 2006 cognitive assessment units, because of limited validity and worries about negative

impact on achievement. However, classical questionnaire scales covering student interest (investigating

two aspects: science in general, and science as experienced in school) and instrumental motivation will be

developed. To ensure cross-cultural validity, and control for response bias, anchoring vignettes will be

provided.

47. While PISA 2006 implemented several highly correlated scales for student beliefs on

environmental issues, only one scale for “environmental awareness” will be kept in PISA 2015. Instead,

PISA 2015 will address more general student views on science, especially – as explicated in the PISA 2015

Science Framework – “Valuing scientific approaches to enquiry”. Another innovation in the PISA 2015

science framework is the treatment of technology as a substantial and integral aspect of science literacy,

rather than a matter of “application”. The Student Questionnaire reflects this change by asking about

frequency of, commitment to, beliefs about, and perceived competence in use of technology. Finally, the

2006 measure of science-related self-concept will be taken up, now complemented by self-efficacy – a

construct that in PISA 2012 was shown to be strongly related to student performance.

Module 7: Student SES, family and home background

48. In order to compare equity related to social and ethnic factors across PISA cycles, the PISA

consortium (Core 6) staff intend to keep measures of socio-economic status and other background

variables basically unchanged. However, the measures of home possessions will be updated, experience

with the new international coding system for occupations (ISCO) from PISA 2012 will be reviewed, and

the new international system of educational levels (ISCED) will be applied in PISA 2015. Furthermore,

alternative ways of asking about parental occupations in the Field Trial will be developed, because the

quality of these data may change when students need to type into a computer.

Module 9: Educational pathways in early childhood, Module 14: Parental involvement, and the Parent

Questionnaire option

49. When children enter primary school, they already differ in their language, pre-reading, and early

numeracy skills and these differences are often maintained later in life. Promoting school readiness and

better adjustment to school is hypothesised to be an efficient means of raising the achievement levels of all

children, but especially of those children who experience a lack of parental support or who grow up in

EDU/PISA/GB(2012)28

14

disadvantaged circumstances. It has been argued that investing in early education programmes will have

large long-term monetary and non-monetary benefits. Early childhood education has been included in three

PISA cycles so far. The importance of pre-school quality has been acknowledged and analysed by OECD

reporting as well.

50. There are good reasons to gather more information on students' early educational pathways in

PISA. However, research on biographical memory suggests that regarding the most important aspects, e.g.

type, quality, or dosage of early child care, no reliable data will be gleaned from 15-year-olds answering

retrospective questions about educational experiences they made a decade or more ago. So, while keeping

a short question on ISCED 0 attendance in the Student Questionnaire, PISA consortium (Core 6) staff –

following advice from QEG and external experts – intends to implement a series of questions in the parent

questionnaire, expecting parents to be the more reliable source of information. Those countries applying

the optional parent questionnaire will gain information on basic characteristics of the early childhood

education and care arrangements of PISA participants and reasons for attending or not attending early

childhood education and care.

51. Over the past years, the involvement of parents in educational processes has gained importance in

the educational debate, and to some extent it has also become relevant for educational policy. Parents are

not only an important audience, but powerful stakeholders in education, thus information on parents’

opinions and engagement is highly valuable also for large scale assessments like PISA. PISA already

covers important aspects of parental involvement, mostly addressed in the international questionnaire

option of the “Parent Questionnaire”. In countries participating in this option, the parents of the students

sampled for PISA provide information on their engagement that focuses on the interaction with the

institutional learning setting, i.e. teachers and schools, and their engagement that focuses on the interaction

with the student, i.e. learning support. Aspects that are covered in this questionnaire include home to

school collaboration, the learning support parents provide to their child, the availability of home resources

for learning as well as parents’ attitudes and beliefs. In PISA 2015, the aspects of home to school

collaboration and parental learning support shall be covered in more depth than in previous cycles by

adding new constructs to the parent questionnaires.

52. Independently from the parent questionnaire option, aspects of parental involvement are

addressed in other questionnaires: Teachers will be asked about their collaboration with parents, for

example through scheduled parent-teacher meetings. More importantly – because this will allow the topic

to be covered in all participating countries -, a question from PISA 2012 has been adapted to address

parental influence on students’ views of science, and new items on parental support have been developed.

Following approaches that have been successfully implemented in research, several aspects of parental

support can be discriminated: stimulation/scaffolding, emotional warmth, guidance and boundaries, and

stability. Finally, school administrators will be asked about school policies on home to school collaboration

implemented in their schools. This includes the aspect of informing parents about their child’s progress,

but also different forms of enhancing parents’ engagement in school activities.

Module 10: Domain-general student behaviour and attitudes

53. Domain-general student behaviour and attitudes refer to the set of student factors or constructs

that cut across curricular topics, or that are independent of curricula, but that are nevertheless important for

and reflective of success in education. Alternative labels that have been used in the literature are “non-

cognitive skills”, “non-intellectual correlates of GPA”, or “21st Century Skills”. Recently economic

models have included this information to better understand labour market outcomes. Meta-analyses show

that these characteristics predict educational success to a degree that is comparable to cognitive ability

measures.

EDU/PISA/GB(2012)28

15

54. So far, ICT-related skills and attitudes, and general attitudes towards the school (including “sense

of belonging”, a measure of well-being in school) have been the only types of domain-general outcomes

addressed in PISA. The former will be taken up in the optional ICT Questionnaire, while the latter will be

included in the Student Questionnaire, providing trend information relating to previous PISA assessments.

PISA 2012 added an assessment of dropout rates – a kind of negative outcome – by school administrators,

and an assessment of some basic student preferences related to problem solving, called “industriousness”

and “openness” in the psychological literature. Also, academic self-efficacy was assessed as a very general

indicator of self-related beliefs. PISA consortium (Core 6) intends to broaden the array of student

preferences and self-related beliefs, and provide anchoring vignettes to establish cross-cultural validity. For

example, grade-related expectations are added as a new aspect of self-related beliefs, and a link to Module

19 (student assessment), which is also a high priority area in PISA 2015.

55. All non-cognitive outcomes mentioned so far are conceptually related to student achievement,

learning and schooling. However, with support from PGB and QEG, the PISA consortium (Core 6)

intends to move into broader areas of non-cognitive outcomes, namely general well-being, health, and time

use. While student behaviour and attitudes have generally been rated very high in policy priority, there

may be some doubts as to the feasibility and relevance of these specific constructs. The QEG, at its recent

meeting, acknowledged these cautions, but advised to apply these questions in the Field Trial to see how

students react, and how the respective data relates to other student, family, and school related factors. Are

well-being and health related to student achievement? Is this relationship shaped by family background

and/or school policies and school climate? Addressing such questions for analysis and reporting should be

highly relevant for educational policy making. The time use data are important for Module 12 (learning

time) as well. This question has been simplified, shortened, and focused on learning activities; it is now a

very innovative computer-based approach to the assessment of learning activities.

Module 12: Learning Time and Curriculum, and the Educational Career optional questionnaire

56. Students’ educational achievement is influenced by a wide array of opportunities to learn. On a

very broad level of abstraction, students’ opportunities to learn can roughly be described by the quantity

and the quality of learning instances. As outlined above, Module 2 will predominantly introduce concepts

of quality of education and teaching in PISA 2015 and will thus cover the quality of learning opportunities.

Module 12 and the Educational Career optional questionnaire (ECQ) then complement Module 2 by

focussing on the current quantity of learning experiences. While these constructs have been part of most

previous PISA cycles, PISA 2015 will deliver critical innovations.

57. The learning time and curriculum experienced by students in their educational career are closely

related to student outcome. On a global level, learning time and curriculum are system characteristics, yet

the intended learning opportunities need to be realised on school and on classroom level, and finally

students must avail themselves of learning time and curriculum in order to learn. A number of factors

influence this implementation in schools and in the classroom, e.g. resources such as rooms, media

devices, and textbooks as well as staff. Also all levels involved in providing and using learning

opportunities hold the possibility to lose a share of the intended learning time and learning opportunities.

This happens when schools close unscheduled (e.g. due to bad weather or safety issues) or need to

reschedule due to poor resources or problematic teacher and student behaviour; on the individual student’s

part there remain problems of truancy and inattentiveness. Taking into account the different perspectives of

intended learning opportunities on the one hand (i.e. learning time provided and intended curriculum) and

loss of opportunities due to difficulties in realisation and implementation on system, school, classroom, and

individual levels on the other hand, PISA 2015 will provide more detailed and more valid information on

learning time than previous cycles. This approach to measuring learning time as it is offered, implemented,

used or not used was highly appreciated by the QEG; its feasibility will be checked in the Field Trial.

EDU/PISA/GB(2012)28

16

58. Besides allowing for a detailed comparison of school systems on country level, the proposed

questionnaire components provide valuable indicators to compare educational opportunities across

different subgroups of students. Social or ethnic disparities in access to learning opportunities can thus be

analysed. The approach used in Module 12 will enable differentiation between disparities in provision and

use of educational opportunities.

59. Students’ learning does not only relate to education in schools alone. Achievement level at the

age of fifteen is determined by all synchronic and diachronic learning experiences. In recent years, more

and more non-school learning instances have gained importance worldwide. Besides looking only at

opportunities to learn in school, PISA needs to ask for time spent in other kinds of educational settings. In

previous cycles, a certain problem has always been to disentangle regular (mandatory) and additional

lessons, private supplementary lessons and private additional study, in- and out-of-school learning

experiences. Therefore, Module 12 will expand the assessment of learning time, including loss of intended

time, to some out-of-school environments. However, detailed coverage of such context goes beyond the

scope of the standard Student Questionnaire. Therefore, with strong support from the QEG, the PISA

consortium (Core 6) has added a focus on supplementary learning opportunities to the Educational Career

optional questionnaire (EC Option). Module 12 and the EC Option seamlessly connect to each other

providing a basic understanding of intended and used learning opportunities in school and supplementary

to school, in curricular and non-curricular domains. PISA 2015 will therefore provide insight in a large

array of learning instances tapping the most relevant sources of education in the students’ lives.

Module 15: School leadership and management

60. The PISA measure of educational leadership has seen a major revision in the 2012 study. Core 6

will carefully analyse the results and revise the measure if needed. An additional question focusing on

instructional leadership has been recommended by the QEG. The new optional Teacher Questionnaire will

be used to gather information on instructional and transformational leadership from teachers as well,

because research has shown that teachers’ perspective on leadership can differ from the positions held by

school administrators.

61. The PISA 2015 Field Trial will also address specific school policies in various areas of school

management and school improvement. Several new measures are spread across a number of Modules,

assessing policies on learning time and the school curriculum (Module 12), teacher recruitment and

professional development (Module 1), internal as well as external evaluation and teacher accountability

(Module 19), parental involvement (Module 14), and multicultural education (Module 8). Altogether, these

measures allow to draw a multi-faceted picture of key management activities.

Module 19: Assessment, evaluation, and accountability

62. Research has shown that assessment-related policies and practices are among the most influential

processes on the classroom, the school, and the system level. Educational reforms in many countries are

centred on assessment, evaluation and accountability. Instruments like performance standards, standard-

based assessment, centralised exams, internal and external evaluation, annual reports on student progress,

and school inspectorates have been promoted and implemented across continents. Analysing their

relationships with other characteristics of schools can add to our understanding of school policies and

student outcomes. The PISA questionnaires can help to describe assessment practices, to compare them

between countries, and to study their relationship with school level variables.

63. This module has received the highest rating in priority from the PGB members – probably

because the issue is very prominent in OECD reports on PISA, but still weakly represented in the

questionnaire design itself. In the most recent School Questionnaire (PISA 2012), student assessment

EDU/PISA/GB(2012)28

17

practices and their purposes were covered, as well as school evaluation, teacher evaluation, and

accountability policies and practices. The very broad notion of “standardised tests” is used here – whether

these tests are high stakes or low stakes, local or national, etc.

64. Module 19 has been designed to draw a more sophisticated picture. Therefore, the module

systematically discriminates student assessment, which mostly takes place in classrooms, from evaluation,

which takes place at the school level. For both levels, information is gathered on (a) frequency of using

different assessment practices, (b) purposes and consequences of the assessment. While the questions on

student assessment focus on formative assessment and feedback (also called “Assessment for learning”),

the questions on evaluation mainly deal with internal rather than external evaluation, because research has

shown formative assessment and internal evaluation to have larger impact on student learning and school

development than summative assessment and external evaluation, respectively. Information from different

perspectives (student, teachers, administrators) will be combined. Results will also be important for

Modules 2 (teaching) and 15 (school leadership and management).

3. Questionnaire design and rotation in the Main Study StQ

3.1 Designing the PISA 2015 Field Trial

65. The full material which is proposed for the Field Trial is documented in the Appendix. Some

further changes to this material can be expected based on:

cognitive labs and focus group studies which are currently underway,

feedback from QEG (at its virtual meetings in December, 2012), from NPMs (through the web-

based forum which will open in January, 2013), from the Secretariat, and of course from the

PGB,

and further revisions made by the PISA consortium (Core 6) staff, questionnaire developers and

contracted experts.

66. A number of single items or questions may be changed or dropped as a result of these

deliberations. However, the overarching modular design of the questionnaires – as explained in section 1 of

this paper - and the framework of policy issues and constructs developed over the past year – as elaborated

in section 2 – will remain unchanged. Development of the constructs listed in the appendix has been based

on policy priorities defined by the participating countries, results from previous PISA studies, and an

extensive review of relevant research literature.

67. The material documented in the appendix altogether covers an estimated assessment time of 97

minutes for the FT Student Questionnaire, 55 minutes for the FT School Questionnaire, 27 minutes for the

FT Parent Questionnaire, 60 minutes for the FT Teacher Questionnaire, and about 18 minutes each for the

FT Educational Career Questionnaire and 14 minutes for the FT ICT Familiarity Questionnaire. In all

cases, the amount of material adheres to the time estimates for Field Trial instruments outlined in the

Integrated Design (see Document submitted by Core 3 and Core 6). This is possible because the Field Trial

design – in line with practices established in all previous PISA studies – will allow for different booklets

randomly being allocated to subgroups of students (for Student Questionnaire, Educational Career

Questionnaire option, and ICT Questionnaire option) and Teachers (for the Teacher Questionnaire option).

No person will have to sit on the questionnaire above time limits.

68. By field trialling this material, the following goals of questionnaire development in PISA can be

met:

EDU/PISA/GB(2012)28

18

allowing for broad coverage of topics in all 19 modules;

assessing high priority modules with more material to provide deeper insights;

establishing close fit to the PISA 2015 major domain of assessment, namely science;

reviewing and - if feasible - reusing or revising any relevant material from earlier PISA cycles,

especially from PISA 2006, (a) to make efficient use of earlier translation and adaptation efforts,

and (b) to establish trend in reporting on context variables and non-cognitive outcomes;

trying out new, innovative questions and question formats – including interactive formats,

context-dependent questions, and formats intended to secure cross-cultural comparability, such as

anchoring vignettes;

ensuring feasibility of computer-based delivery for all previously established material.

69. The breadth of Field Trial material will allow for selecting the most appropriate content for the

Main Study. Introducing new, innovative content and format always bears the risk of not working on a

large scale, even if these innovations are based on sound research experience. This especially holds true

when a new mode of delivery (here: computer-based assessment) is being introduced at the same time.

From a policy point of view, using larger amounts of material in the Field Trial provides more degrees of

freedom for policy makers (PGB) when selecting Main Study content based on the Field Trial data.

3.2 Time restrictions and design implications for the Main Study

70. The Main Study will see much stronger time restrictions than the Field Trial (see Integrated

Design): approximately 30 minutes for the Student Questionnaire, 45 minutes for the School

Questionnaire, 30 minutes each for the Teacher and the Parent Questionnaire, and 10 minutes each for the

Educational Career and the ICT options. Thus, severe cuts will be made from the Field Trial to the Main

Study, especially regarding the two standard instruments, School and Student Questionnaire.

71. In preparation for this reduction exercise, the amount of material that might be taken up from

previous cycles has been estimated (cf. footnote in the Appendix). This hypothetical selection was based

on the PGB’s priorities, discussion at the QEG and a review of existing PISA reports (as in Table 1 above).

This selection of well established questions amounts to about 30 minutes for the Student Questionnaire,

and about 22 minutes in the School Questionnaire. Regarding the School Questionnaire, there is ample

space left for new, innovative content. Regarding the Student Questionnaire, however, these estimates

underpin the importance of finding ways of optimising testing time. Otherwise, the PISA 2015 Student

Questionnaire would be left without any innovation and improvement; neither in content nor in measures.

72. PISA 2012 introduced a rotated booklet design for the student questionnaire, with an eight

minutes “core” set of questions covering student and family background (in our terminology: Modules 7

and 8), while all other content was given to 2/3 of the students only. An estimated 33 minutes of non-core

content were formed into three clusters (A, B, C), and each student was randomly assigned to a

combination of two of these clusters (AB, BC, or CA). Random assignment of students to booklets ensured

that parameters reported for questionnaire constructs – like means, standard deviations, and percentiles –

would be unbiased. Altogether, the rotation scheme allowed for coverage of approximately 40 minutes of

estimated assessment time in the Student Questionnaire.

73. The test design has become more complex in PISA 2015. The computer-based tests will assess

four domains (science, reading, mathematics, and collaborative problem solving), while most individual

EDU/PISA/GB(2012)28

19

students will be working on two of these domains only. These settings require more student scores (namely

50 per cent of all student scores) to be estimated (“imputed”) from achievement in other domains and

background information, thus putting high burden on the background information produced from

questionnaires. The suitability of a rotated design as used in PISA 2012 has been called into question

because of this increase in complexity for the assessment in PISA 2015.

74. Therefore the recommendation from the PISA 2015 Consortium and the QEG is that there should

be only one version of the student questionnaire in the PISA 2015 main study and no rotation, if enough

innovative material can be included. The results of the field trial would be used - taking advantage of the

more accurate information on timing which will be obtained from the computer-based administration - to

determine how much time will be needed to cover both established and innovative content in the main

study.. Furthermore, the time available for the administration of the student questionnaire in the main study

for 2015 might need to be revised based on the results of the field trial. The analysis of PISA 2012 data,

especially a careful evaluation of the rotation design applied, will also be considered when determining the

design of the background questionnaires in the PISA 2015 main study.

EDU/PISA/GB(2012)28

20

REFERENCES

Almlund M., A.L. Duckworth, J.J. Heckman and T. Kautz (2011), "Personality Psychology and

Economics," IZA Discussion Papers 5500, Institute for the Study of Labour (IZA), Bonn.

Barnett, W.S. (2008), Preschool Education and its Lasting Effects: Research and Policy Implications,

Education and the Public Interest Center & Education Policy Research Unit, Boulder and Tempe,

http://epicpolicy.org/publication/preschooleducation, retrieved 27 September 2012.

Blau D. and J. Curie (2006), "Pre-School, Day Care, and After-School Care: Who's Minding the Kids?" in

Handbook of the Economics of Education, Elsevier, North Holland, pp. 1164-1278.

Bryk, A.S., P. Sebring, E. Allensworth, S. Luppescu and J. Easton (2010), Organizing School for

Improvement, Lessons from Chicago, The University of Chicago.

Creemers, B.P.M. and L. Kyriakides (2008), The Dynamics of Educational Effectiveness: A Contribution

to Policy, Practice and Theory in Contemporary Schools, Routledge, London.

Cunha, F., J.J. Heckman, L.J. Lochner and D.V. Masterov (2006), "Interpreting the Evidence on Life Cycle

Skill Formation,", in E. Hanushek and F. Welch (eds.), Handbook of the Economics of Education,

Elsevier, North Holland, pp. 697-812.

Hanushek, E.A. and L. Wößmann (2011), “The Economics of International Differences in Educational

Achievement”, in E.A. Hanushek, S. Machin and L. Wößmann (eds.), Handbook of the Economics

of Education, Vol. 3, North Holland, Amsterdam, pp. 89–200.

Heckman J.J., J. Stixrud and S.Urzua (2006), "The Effects of Cognitive and Noncognitive Abilities on

Labor Market Outcomes and Social Behavior", Journal of Labor Economics, University of Chicago

Press, Vol. 24, 3, pp. 411-482.

OECD (2007), Science Competencies for Tomorrow's World, Executive Summary, OECD Publishing,

Paris.

OECD (2009),” PISA 2009 Questionnaire Framework”, in PISA 2009 Assessment Framework: Key

Competencies in Reading, Mathematics and Science, OECD Publishing, Paris, pp. 149-169.

OECD (2010), PISA 2009 Results: What Students can do? Student Performance in Reading, Mathematics

and Science, Vol. 1, OECD Publication, Paris.

Sammons, P., K. Sylva, E. Melhuish, I. Siraj-Blatchford, B. Taggart, K. Toth, D. Draghici. and R. Smees

(2011), Effective Pre-School, Primary and Secondary Education Project (EPPSE 3-14): Influences

on Students’ Attainment and Progress in Key Stage 3: Academic Outcomes in English, Maths and

Science in Year 9, Institute of Education, University of London / Department for Education, London.

EDU/PISA/GB(2012)28

21

Scheerens, J. and R.J. Bosker (1997). The Foundations of Educational Effectiveness, Pergamon Press,

Oxford.

Sykes G., B. Schneider and D.N. Plank (eds.) (2009), Handbook of Educational Policy Research,

Routledge, New York.

Wößmann, L., E. Lüdemann, G. Schütz and M.R. West (2007), “School Accountability, Autonomy, Choice,

and the Level of Student Achievement: International Evidence from PISA 2003”, OECD Education

Working Paper No. 13.

EDU/PISA/GB(2012)28

22

APPENDIX: DRAFT OVERVIEW OF FIELD TRIAL CONTENT

Module Policy issue and construct area Specific construct to be

measured Status of the measure

Estimated

Time

(sec)

Instrument

1 Teacher qualification and professional knowledge

1 Teacher background Gender new to PISA, from TALIS 2013 10 TQ Core

1 Teacher background Age new to PISA, from TALIS 2013 10 TQ Core

1 Initial education Level of education new to PISA 15 TQ Core

1 Initial education

Teacher education or training

programme: completion, mode of

qualification, and duration

new to PISA 60 TQ Core

1 Initial education Subjects studied, subjects

teaching

new to PISA 100 TQ Core

1 Teacher background Workplace selection new to PISA 120 TQ Core

1 Teacher background Employment status new to PISA, from TALIS 2013 10 TQ Core

1 Teacher background Job experience – number of

schools new to PISA 10 TQ Core

1 Teacher background Job experience – number of years new to PISA, from TALIS 2013 20 TQ Core

1 Teacher background Teaching modal grade new to PISA 10 TQ Core

1 Professional development Professional development

activities new to PISA 110 TQ Core

1 Professional development

Obligation to participate in

professional development

activities

new to PISA 10 TQ Core

1 Professional development Support for professional

development activities new to PISA 20 TQ Core

1 Professional development

Content distribution for initial

Education and professional

development Activities

new to PISA 140 TQ General

EDU/PISA/GB(2012)28

23

1 Professional development Teacher cooperation - general new to PISA 80 TQ General

1 Professional knowledge and teacher

beliefs

Self-efficacy in classroom

management new to PISA 40 TQ General

1 Professional knowledge and teacher

beliefs

Self-efficacy in maintaining

positive relations with students new to PISA 40 TQ General

1 Professional knowledge and teacher

beliefs Performance vs. needs orientation new to PISA 60 TQ General

1 Non-cognitive factors Job satisfaction new to PISA 100 TQ General

1 Initial education Content distribution for initial

education new to PISA 50 TQ Science

1 Professional development Science specific teacher

collaboration new to PISA 220 TQ Science

1 Professional development

Content distribution for

professional development

activities

new to PISA 50 TQ Science

1 Professional knowledge and teacher

beliefs

Self-efficacy related to science

content new to PISA 80 TQ Science

1 Professional knowledge and teacher

beliefs

Self-efficacy related to teaching

science content new to PISA 80 TQ Science

1 Non-cognitive factors Teacher enthusiasm (vignettes) new to PISA 80 TQ Science

1 Non-cognitive factors Teacher enthusiasm (scales) new to PISA 190 TQ Science

1 Initial education Initial education of all teaching

staff

previous (2000) (2003) (2006)

(2009) (2012) 120

* ScQ

1 Initial education Requirement for teacher

education or training programme new to PISA 20 ScQ

1 Initial education Teacher recruitment policies new to PISA 20 ScQ

1 Professional development Professional development policy new to PISA 150 ScQ

1 Professional development Participation in professional

development 2000, (2012) 20 ScQ

1 Non-cognitive factors Perceived teacher engagement new to PISA 70 StQ

2 Science teaching practices

2 Teaching practices

Science teaching: Inquiry-based –

(4 dimensions: procedural,

epistemic, social, and real life

applications)

previous (2006) and extended 130* StQ

2 Teaching practices Science teaching: direct

instruction changed (2012) 70 StQ

2 Teaching practices Science teaching: Opportunities new to PISA 120 StQ

EDU/PISA/GB(2012)28

24

to learn

2 Teaching quality Teacher support previous (2000, 2003, 2012) 50* StQ

2 Teaching quality Perceived support for competence new to PISA 40 StQ

2 Teaching quality Perceived support of autonomy new to PISA 40 StQ

2 Teaching quality Disciplinary climate previous (2000, 2003, 2009, 2012) 50* StQ

2 Teaching quality Cognitive activation changed (2012) 70 StQ

2 Teacher perception Concepts of inquiry based science

teaching new to PISA 160 TQ Science

2 Teacher perception Science teaching practices new to PISA 120 TQ Science

3

School learning environment for science

(For other aspects of the school environment, see modules 8, 12, 13,

15, and-19)

3 Science teaching staff Number of teachers per science

area new to PISA 120 ScQ

3 Policy and actions taken to improve

teaching/

Value of science education in

school community new to PISA 80 ScQ

3 Policy and actions taken to improve

teaching/ Science-related collaborations new to PISA 40 ScQ

3 Policy and actions taken to improve

teaching/ Science-specific resources new to PISA 60 ScQ

4 Science-related outcomes: attitudes, motivation, value beliefs,

strategies

4 Motivation Interest in school science subjects changed (2006) 40* StQ

4 Motivation Interest in broad science topics changed (2006) 160* StQ

4 Motivation Anchoring Vignette Interest in

Science 60 StQ

4 Motivation Instrumental motivation changed (2006), previous (2012) 40* StQ

4 Beliefs about science General and personal value of

science previous (2006) 100 StQ

4 Beliefs about science Occupational prestige new to PISA 80 StQ

4 Beliefs about science Dealing with uncertainty and

ambiguity new to PISA 100 StQ

4 Beliefs about science Valuing scientific approaches to

enquiry new to PISA 80 StQ

4 Beliefs about science Students' view on science new to PISA 40 StQ

4 Self-related cognitions Self-Concept previous (2006) 50 StQ

4 Self-related cognitions Self-Efficacy new to PISA 80 StQ

4 Use of technology Use of technology: frequency new to PISA 90 StQ

EDU/PISA/GB(2012)28

25

4 Use of technology Use of technology: perceived

competence new to PISA 50 StQ

4 Use of technology Use of technology: commitment new to PISA 100 StQ

4 Use of technology Use of technology: beliefs new to PISA 100 StQ

4 Environmental awareness Awareness of environmental

issues previous (2006) 70

* StQ

5 Out-of-school science experience

(see also module 12)

5 Out-of-school science experience Leisure time science activities

media related changed (2006) 60

* StQ

5 Out-of-school science experience Extra-curricular science activities

in school previous (2006) 110 ScQ

5 Out-of-school science experience Science activities at age 10 previous (2006) 50 PQ

6 Science career

6 Career aspiration Broad science aspiration previous (2006) 40* StQ

6 Career aspiration Specific career aspiration at age

30 previous (2006) 20 StQ

6 Students own preparation Student information on science

careers previous (2006) 40 StQ

7 Student SES, family and home background

7 Student background Grade PISA 2012 10* StQ

7 Student background Educational programme PISA 2012 10* StQ

7 Student background Age PISA 2012 10* StQ

7 Student background Sex PISA 2012 10* StQ

7 Family background Household members PISA 2012 40* StQ

7 Parental background Mother's profession PISA 2012 40* StQ

7 Parental background Mother's school education PISA 2012 20* StQ

7 Parental background Mother's tertiary education PISA 2012 20* StQ

7 Parental background Mother's employment status PISA 2012 20* StQ

7 Parental background Father's profession PISA 2012 40* StQ

7 Parental background Father's school education PISA 2012 20* StQ

7 Parental background Father's tertiary education PISA 2012 20* StQ

7 Parental background Father's employment status PISA 2012 20* StQ

7 Migration background Migration background PISA 2012 30* StQ

7 Migration background Age at immigration PISA 2012 10* StQ

7 Migration background Language spoken at home PISA 2012 30* StQ

7 Family background Home possessions - existence PISA 2012 120* StQ

7 Family background Home possessions - number PISA 2012 50* StQ

EDU/PISA/GB(2012)28

26

7 Family background Number of books PISA 2012 20* StQ

7 Parental background Number of children in household previous (2009) 7 PQ

7 Parental background Age of parents previous (2006, 2012) 10 PQ

7 Parental background Parents' qualification (ISCED) previous (2006, 2009, 2012) 20 PQ

7 Parental background Parental occupation previous (2006, 2012) 60 PQ

7 Parents' background Spending on educational services previous (2006, 2009, 2012) 40 PQ

7 Parents' background Household income previous (2006, 2009, 2012) 15 PQ

7 Parents' background Home resources previous (2009) 20 PQ

7 ICT Availability of ICT at home previous (2009, 2012) 60 ICT

7 ICT General use of ICT previous (2012) 50 ICT

7 ICT Use of ICT outside of school (for

leisure activities) previous (2012) 60 ICT

7 ICT Use of ICT outside of school (for

school work activities) previous (2012) 60 ICT

8 Ethnicity and Migration

8 Ethnicity and Migration Cultural diversity of school new to PISA 30 TQ General

8

Multicultural school

culture/Concrete policies and

practices

Multicultural education new to PISA 50 TQ General

8

Multicultural school

culture/Concrete policies and

practices

Multicultural education new to PISA 50 ScQ

8 Multicultural school culture Basic assumptions on diversity new to PISA 80 ScQ

8 Multicultural school culture Basic assumptions on diversity new to PISA 80 TQ General

8 Multicultural school culture School intercultural climate new to PISA 60 StQ

8 Perceived discrimination in society Membership in a minority group new to PISA 20 StQ

8 Perceived discrimination in society Perceived reasons for

discrimination new to PISA 30 StQ

8 Migration background Language preferences for

communication previous (2012) 40 StQ

8 Parental involvement Barriers to parental involvement new to PISA 60 PQ

8 Parent’s migration background Country of birth previous (2012) 60 PQ

8 Parent’s migration background Citizenship previous (2012) 20 PQ

9 Educational pathways in early childhood (and thereafter)

9 Prior to school entry ISCED 0 attendance previous (2003), (2009), (2012) 10* StQ

9 Education biography School enrolment previous (2003), (2009), (2012) 10* StQ

9 After school entry Repeating grades previous (2003), (2009), (2012) 30* StQ

9 After school entry Skipping grades new to PISA 30 StQ

EDU/PISA/GB(2012)28

27

9 School enrolment School enrolment relative to

mandatory school starting age new to PISA 30 PQ

9 School entry Age at school entry new to PISA 10 PQ

9 Prior to school entry Educational biography new to PISA 70 PQ

9 Education biography ECEC attendance prior to ISCED

1 new to PISA 10 PQ

9 Education biography Age at first ECEC attendance new to PISA 10 PQ

9 Education biography Type of predominant ECEC

arrangement new to PISA 30 PQ

9 Education biography ECEC dosage new to PISA 10 PQ

9 Education biography ECEC type of care arrangement new to PISA 20 PQ

9 Education biography ECEC reasons for attendance new to PISA 40 PQ

9 Education biography ECEC reasons for not attending new to PISA 80 PQ

9 Education biography Reasons for ECEC setting choice new to PISA 100 PQ

9 Education biography Change of school in ISCED 1 previous (2003, 2009) 15 EC

9 Education biography Change of school in ISCED 2 previous (2003, 2009) 20 EC

9 Education biography Change of school in ISCED 3 previous (2003, 2009) 20 EC

9 Education biography Change of study program in

ISCED 2 previous (2003) 20 EC

9 Education biography Change of study program in

ISCED 3 previous (2003) 20 EC

10 Domain-general student behaviour and attitudes

10 Student preferences and study skills Perseverance previous (2012) 30* StQ

10 Student preferences and study skills Industriousness new to PISA 30 StQ

10 Student preferences and study skills Organization new to PISA 60 StQ

10 Student preferences and study skills Procrastination new to PISA 60 StQ

10 Student preferences and study skills Openness (for Problem Solving) previous (2012) 50* StQ

10 Student preferences and study skills Anchoring Vignettes for student

preferences new to PISA 90 StQ

10 Self-related beliefs Test Anxiety new to PISA 60 StQ

10 Self-related beliefs Strategic Approach to Learning new to PISA 50 StQ

10 Self-related beliefs Academic Self Efficacy previous (2012) 30* StQ

10 Self-related beliefs Performance Self Efficacy new to PISA 30 StQ

10 Self-related beliefs Anchoring vignettes for self-

related scales new to PISA 120 StQ

10 Self-related beliefs Grade Goal and current grades new to PISA 15 StQ

10 Attitudes towards school Attitudes towards School -

Learning Activities previous (2012) 40

* StQ

EDU/PISA/GB(2012)28

28

10 Attitudes towards school Attitudes towards School -

Learning Outcomes previous (2012) 30

* StQ

10 well-being General Life Evaluation new to PISA 60 StQ

10 well-being Life Satisfaction new to PISA 50 StQ

10 well-being Anchoring vignettes new to PISA 60 StQ

10 well-being Well-being at school/Sense of

belonging previous (2012) 90

* StQ

10 Health Over-/Underweight new to PISA 30 StQ

10 Health Physical Activities new to PISA 40 StQ

10 Health Nutrition new to PISA 60 StQ

10 Health Health-related school practices new to PISA 120 StQ

10 time use Sleeping Time new to PISA 20 StQ

10 time use Time spent at school new to PISA 20 StQ

10 time use

shortened day reconstruction time

protocol focused on

study/learning-related activities

new to PISA 200 StQ

10 Idea generation Consequences new to PISA 180 StQ

10 School dropout Number of dropouts previous (2012) 20* ScQ

10 ICT Engagement ICT Interest new to PISA 120 ICT

10 ICT Engagement Perceived ICT Competence new to PISA 120 ICT

10 ICT Engagement Perceived Autonomy related to

ICT Use new to PISA 120 ICT

10 ICT Engagement ICT as a topic in Social

Interaction new to PISA 120 ICT

11 Student dispositions related to collaborative problem solving

11 Collaboration and teamwork

dispositions

Teamwork dispositions:

Cooperate, guide and negotiate new to PISA 100 StQ

11 Experience related to collaborative

problem solving

In-school experience in

collaboration-related activities new to PISA 55 StQ

11 Experience related to collaborative

problem solving

Out-of-school experience in

collaboration-related activities new to PISA 25 StQ

12 Learning time and curriculum

12 Allocated learning time Instructional weeks per year new to PISA 20 ScQ

12 Allocated learning time Instructional days per year new to PISA 20 ScQ

12 Loss of learning time Loss on national/regional level new to PISA 40 ScQ

12 Loss of learning time Loss on school level new to PISA 60 ScQ

12 Provided additional learning time Additional learning time,

availability previous (2012) 10

* ScQ

EDU/PISA/GB(2012)28

29

12 Provided additional learning time Additional learning time, purpose previous (2012) 30 ScQ

12 Provided additional learning time Additional other study support previous (2000) 20 ScQ

12 School policies School policies on additional

lessons new to PISA 10 ScQ

12 School policies School policies on additional

tutoring lessons content new to PISA 30 ScQ

12 School policies Handling of additional tutoring in

the school/school district new to PISA 80 ScQ

12 Students’ learning time Number of school lessons in

maths, science, and test language previous (2000-2012) 30

* StQ

12 Students’ learning time Learning time per science subject new to PISA 100 StQ

12 Students’ learning time All lessons per week previous (2012) 10* StQ

12 Students’ learning time Minutes per lesson previous (2012) 30* StQ

12 Loss of learning time Loss on classroom level new to PISA 10 StQ

12 Loss of learning time

Loss on individual level in

regular school lessons: within

lessons, whole lessons, full

school days

new to PISA 60 StQ

12 Loss of learning time Reasons for loss on individual

level new to PISA 10 StQ

12 Students’ additional learning time Amount of additional lessons and

study time previous (2000-2012) 180

* StQ

12 Students’ additional learning time

Loss on individual level in

regular school lessons: within

lessons, whole lessons, full

school days

new to PISA 50 StQ

12 Intended Science Curriculum Content: Components described

in the science curriculum new to PISA 30 TQ Science

12 Intended Science Curriculum Content: Approaches and

processes new to PISA 80 TQ Science

12 Curriculum coherence Transfer across classes and grades new to PISA 30 TQ Science

12 Textbook use Focus of curriculum content new to PISA 20 TQ Science

12 Realized curriculum Science courses taken new to PISA 50 StQ

12 Curriculum choice Choice in science courses new to PISA 20 StQ

12 Additional learning time Number of hours previous (2006) 180 EC

12 Additional learning time Content of additional lessons new to PISA 10 each EC

12 Additional learning time Type of additional lessons new to PISA 60 EC

12 Additional learning time Location of additional lessons new to PISA 50 EC

EDU/PISA/GB(2012)28

30

12 Additional learning time Teacher in additional lessons new to PISA 40 EC

12 Comparing regular school and

additional lessons

Cognitive stimulation, adaptivity

of teaching new to PISA 150 EC

12 Comparing regular school and

additional lessons

Structuring content and classroom

management new to PISA 100 EC

12 Comparing regular school and

additional lessons Teacher-student relationship new to PISA 50 EC

12 Additional learning time Reasons for additional lessons new to PISA 80 EC

12 Additional learning time Reasons for no additional lessons new to PISA 100 EC

12 Education biography History of private supplementary

tutoring new to PISA 30 EC

13

School climate: Interpersonal relations, trust, expectations

(see also modules 2 for classroom climate and 14 for parental

involvement)

13 Teachers and students Student teacher relation at school previous (2000 ff.) with new items 115* StQ

13 Teachers and students Achievement pressure by teachers previous (2000) 40 StQ

13 Students Bullying new to PISA 70 StQ

13 Teacher and students Fairness new to PISA 60 StQ

13 Staff Teacher morale previous (2000, 2003, 2012) 40* ScQ

13 Staff/Students Teacher and student behaviour

affecting school climate previous (2012) 190

* ScQ

13 Teachers and students Student-teacher relation at school adapted from previous StQ 90 TQ

General

14 Parental involvement

14 Parental background Respondent previous (2006, 2009, 2012) 5 PQ

14 Parents' view on science Parents' view on science skills in

the job market previous (2006, 2012) 35 PQ

14 Parents' view on science Parent and peer influence in

student's view of science changed (2012) 60 StQ

14 School choice Perceived school quality previous (2006, 2009, 2012) 45 PQ

14 Home-to-school-collaboration

Parental engagement that focuses

on interaction with the

institutional learning setting

previous (2009, 2012) and new to

PISA 80 PQ

14 Home-to-school-collaboration Topics of parent-teacher

communication new to PISA 60 PQ

14 Home-to-school-collaboration Modes of parent-teacher

communication new to PISA 25 PQ

14 Home-to-school-collaboration Parents' satisfaction with school previous (2009) 120 PQ

EDU/PISA/GB(2012)28

31

14 Home-to-school-collaboration Parents'motivation for

participation new to PISA 45 PQ

14 Home-to-school-collaboration Parents' social relationship and

involvement new to PISA 25 PQ

14 Home-to-school-collaboration

Parental engagement that focuses

on interaction with the

institutional learning setting

previous (2012) 80 ScQ

14 Home-to-school-collaboration Parental expectations towards

school previous (2006, 2009, 2012) 30 ScQ

14 Home-to-school-collaboration School policies for parental

involvement new to PISA 70 ScQ

14 Home-to-school-collaboration Topics of parent-teacher

communication new to PISA 60 TQ General

14 Home-to-school-collaboration Modes of parent-teacher

communication new to PISA 25 TQ General

14 Parental support

Emotional warmth, guidance and

boundaries,

stimulation/scaffolding and

stability

new to PISA 140 PQ

14 Parental support Parental support for learning at

home

previous (2009, 2012) focussing on

reading and mathematics 60 PQ

14 Parental support

Child's past science activities and

parental support for learning at

home

previous (2006) 35 PQ

14 Parental support

Emotional warmth, guidance and

boundaries,

stimulation/scaffolding and

stability

new to PISA 110 StQ

14 Parental support Help with homework or

additional study at home new to PISA 80 EC

14 Parental support Parents’ view on science careers previous (2006, 2012 for

mathematics) 35 EC

15

Leadership and school management

(For specific aspects of school management, see also modules 3, 8,

12, and 16-19)

15 Instructional leadership Leadership tasks new to PISA 50 ScQ

15 School management Educational leadership scale 2009(14), 2012(21) 135* ScQ

15 School policies and practices Curriculum 2012(3) 30 ScQ

EDU/PISA/GB(2012)28

32

15 Use of ICT Resources ICT usage previous (2012) 25* ScQ

15 Transformational leadership Teachers' view on leadership new to PISA 70 TQ General

15 Instructional leadership Teachers' view on leadership new to PISA 110 TQ General

16 Resources

(see also module 3 for science-specific resources)

16 Structure and organisation of the

school Public or private school

2000(1), 2003(1), 2006(1), 2009(1),

2012(1) 10

* ScQ

16 Structure and organisation of the

school

Type of school location

community

2000(1), 2003(1), 2006(1), 2009(1),

2012(1) 15

* ScQ

16 Structure and organisation of the

school Number of students

2000(2), 2003(2), 2006 (2), 2009

(2), 2012(2 15

* ScQ

16 Structure and organisation of the

school Source of resources previous (2006), previous (2012) 40

* ScQ

16 Teaching staff Vacant positions previous (2006) 30 ScQ

16 Class size Average size of classes 2006(1), 2012(1) 11* ScQ

16 ICT ICT-Equipment previous (2012) 30* ScQ

16 Resources Problems due to little resources previous (2012) 130 ScQ

16 ICT Availability of ICT at school previous (2009, 2012) 60 ICT

16 ICT Use of ICT at school previous (2009, 2012) and new to

PISA 90 ICT

17 Locus of decision making within the school system

17 Decision making within the school

system Locus of decision making previous (2006, 2009, 2012) 120* ScQ

18 Admission, selection, and choice

18 Grade retention Policies on grade retention new to PISA 100 ScQ

18 Grouping Grouping policies previous (2006) 20 ScQ

18 Choice Competition between schools previous (2012) 30* ScQ

18 Choice Parental school choice previous (2009), previous (2012) 100 PQ

18 Choice Availability of schools previous (2009, 2012) 25 PQ

19 Admission Student admission policies previous (2000- 2012) 60* ScQ

19 Assessment, evaluation, and accountability

19 Student assessment/Instruments General assessment practice previous (2000, 2003, 2009, 2012) 60* ScQ

19 Student assessment/Instruments Classroom assessment

instruments new to PISA 80 TQ General

19 Student assessment/Instruments Teachers' grading practice new to PISA 170 TQ General

19 Student assessment/Use Adaption of instruction new to PISA 30 StQ

19 Student assessment/Use Adaption on instruction new to PISA 30 TQ Science

19 Student assessment/Use Perceived feedback new to PISA 50 StQ

EDU/PISA/GB(2012)28

33

19 Student assessment/Use Source of feedback new to PISA 90 StQ

19 Student assessment/Use Use of feedback to guide learning new to PISA 60 StQ

19 Student assessment/Use Purpose of assessment results previous (2000, 2003, 2009, 2012) 100* ScQ

19 School evaluation/Norms Evaluation policies previous (2012) 100* ScQ

19 School evaluation/Norms Policies on teacher accountability new to PISA 30 ScQ

19 School evaluation/Instruments Foci of internal evaluation new to PISA 130 ScQ

19 School evaluation/Instruments Foci of internal evaluation new to PISA 130 TQ General

19 School evaluation/Instruments Teacher evaluation previous (2000, 2003, 2009, 2012) 40* ScQ

19 School evaluation/Instruments Teacher evaluation new to PISA 40 TQ General

19 School evaluation/Use Processes on internal evaluation new to PISA 70 ScQ

19 School evaluation/Use Processes on internal evaluation new to PISA 70 TQ General

19 School evaluation/Use Consequences of internal

evaluation new to PISA 130 ScQ

19 School evaluation/Use Consequences of internal

evaluation new to PISA 130 TQ General

19 Accountability Use of achievement data for

accountability previous (2006, 2009, 2012) 30

* ScQ

19 Accountability Teacher incentives previous (2012) 70* TQ General

19 School evaluation/Use Processes on external evaluation new to PISA 50 ScQ

*)

Hypothetical selection of material that might be taken up from previous cycles