Proposed student questionnaire for the main survey
National Research Coordinators Meeting Windsor, June 2008
NRCMeetingWindsor
June 2008
Contents of presentation
• Criteria used for selection of questionnaire material
• Overview of field trial questions and recommendations
• Length of proposed instrument
• Proposed changes
NRCMeetingWindsor
June 2008
Criteria for selection
• Field trial analysis results– Scaling properties (for scaled items)– Item dimensionality– Face validity– Review of relationship between related
variables/constructs
• Coverage of assessment framework• Feasibility of adaptations to national
contexts
NRCMeetingWindsor
June 2008
General proposals
• Only one (combined) student questionnaire (with both background and perceptions questions)
• Only one form
• No “don’t know” categories
NRCMeetingWindsor
June 2008
Review of field trial instruments
• Three categories of questions– To be retained without changes (in green)– To be modified (in yellow)– To be deleted (in red)
• Some questions are flagged as having “medium priority”
• Recent advice from the PAC meeting 7-8 June appears in orange
NRCMeetingWindsor
June 2008
Student characteristics
• Age (A01)
• Gender (A02)
• Ethnicity (optional) (A03)
• Expected educational level (A04)– without “don’t know” category
NRCMeetingWindsor
June 2008
Home environment
• Household composition (B01)– Medium priority only
• Country of birth (B02)
• Language use at home (B03)
NRCMeetingWindsor
June 2008
Parent background
• Parental occupation (B04, B06)– Good measure of socio-economic background– Somewhat larger percentages of missing values
in some countries
• Parental education (B05, B07)– Without “don’t know” category– Still considerable percentages of missing values
• Parental interest in political/social issues (B08)– Interesting results!
NRCMeetingWindsor
June 2008
Household possessions
• Household items (B09)– low reliabilities (especially in more
developed countries)– low correlations with other socio-
economic indicators)– PAC advice: Consider as an
international option
• Books in the home (B10)– Good indicator of socio-economic
background
NRCMeetingWindsor
June 2008
Measuring socio-economic background in the main survey
• It is proposed to have three indicators– Parental occupation (highest of both
parents)– Parental education (highest of both
parents)– Number of books at home
• Analysis will be undertaken to explore computation of composite index
NRCMeetingWindsor
June 2008
Student activities
• Out-of-school leisure time activities (C01)– Keep only items on reading and hanging out– Add: Homework and study time– PAC advice: Consider keeping also (at least)
computer and television use items• Activities out of school (C02)
– Discussion item form scale– Media information items do not scale but are of
interest as single items (wording change for TV news)
– Social participation items do not scale and should not be retained
– PAC advice: Consider combining items on organised social activities (boy scouts, youth clubs etc.)
NRCMeetingWindsor
June 2008
Students’ civic participation
• Civic participation in the community (C03)– Worked as scale and interesting associations
with other variables – PAC advice: An additional note to include
religious services was initially considered but should rather not be included
• Civic participation at school (C04)– Two items reflect different dimension (tutoring,
school newspaper) and are proposed to be deleted
– There were adaptation problems with the item on school protest, which is proposed as another deletion
– PAC suggestion: Consider same categories as for previous question
NRCMeetingWindsor
June 2008
School-related perceptions - 1
• Open climate for classroom discussion (D01)– Good scaling properties and interesting
correlations– PAC advice: Use rather same categories as in
CIVED• Student influence at school (D02)
– Good scaling characteristics but negative correlations with civic knowledge
– ISC sees this as question with medium priority– PAC advice: Additional item about
extracurricular activities
NRCMeetingWindsor
June 2008
School-related perceptions - 2
• School climate items (D03)– Good scaling properties for student-teacher
relation items (medium priority)– Students’ sense of belonging has not always
satisfactory scaling properties and should be deleted
• Students’ confidence in school participation (D04)– Negatively worded items did not load on the
same dimensions and should not be retained
NRCMeetingWindsor
June 2008
Perceptions of citizens and society - 1
• Democratic value beliefs (E01)– Agreement item format leads to highly skewed
responses– Alternative format proposed asked about
characteristics of “ideal society”– PAC advice: Keep original format
• Good citizenship beliefs (E02)– Two-dimensional structure confirmed– Proposed to delete three items that did not scale
well with either of the two scales– PAC advice: Consider keeping “obeying the
law” and “working hard” for single-item reporting
NRCMeetingWindsor
June 2008
Perceptions of citizens and society - 2
• General trust (E03)– Few interesting associations– Only item about trust in “people in
general” should be added to question about trust in institutions
• Acceptance of socially undesirable actions (E04)– Some problems with item scaling
properties and generally not too interesting results
NRCMeetingWindsor
June 2008
Self-perceptions - 1
• Interest in political and social issue (F01)– Generally good scaling properties– Five items to be retained and one modified
• Students’ self-concept (political internal efficacy) (F02)– Six items have good scaling properties– One (negatively phrased) item did not scale well
and should be deleted
NRCMeetingWindsor
June 2008
Self-perceptions - 2
• Feelings of distinctiveness (F03)– Doubts that items measure what is
intended
• Attachment to community levels (F04)– Results not very interesting– May be retained for regional instruments
NRCMeetingWindsor
June 2008
Rights and responsibilities - 1
• Gender rights items (G01)– One item did not scale well and should be
deleted
• Rights for ethnic/racial groups (G02)– Good scaling properties– Correct adaptation need to be ensured
(not minorities!)
NRCMeetingWindsor
June 2008
Rights and responsibilities - 2
• Rights for immigrants (G03)– Negatively phrased items reflect different
dimension and should not be retained– PAC suggestion: Keep item h (“When there
are not many [instead of enough]many [instead of enough] jobs available the number of <immigrants> should be limited“) for single-item reporting
• Reaction to threats to democracy (G04)– Items did not scale and only single-item reporting
might be considered– Proposed to delete this question– PAC advice: Include three items (b, d and f)
in democracy belief question (E01)
NRCMeetingWindsor
June 2008
Perceptions of institutions and society - 1
• Trust in groups and institutions (H01)– Good scaling properties for CIVED items and
interest in single-item reporting for not scaled items
– Addition of “people in general”– “Standardisation” of optional items– PAC suggestion: Consider asking about
media separately (Newspaper, TV, radio, internet) and ask about “people in general” first
• Attitudes toward country (H02)– Generally, good scaling properties except for
three items (two of them should be deleted)
NRCMeetingWindsor
June 2008
Perceptions of institutions and society - 2
• Government responsiveness (external political efficacy) (H03)– Analysis revealed two dimensions
(“responsiveness” and “cynicism”)– Both scales have weak reliabilities
• Satisfaction with country’s achievements (H04)– Generally, good scaling properties– Relatively high correlations with attitudes toward
country– Deletion proposed on content grounds– Mixed PAC advice
NRCMeetingWindsor
June 2008
Perceptions of institutions and society - 3
• Support for political parties (H05)– Interesting results– Sufficient proportions in each category
NRCMeetingWindsor
June 2008
Perceptions of participation in society - 1
• Citizenship self-efficacy (I01)– Good scaling properties for all but one
item (“raising money for people in need”)
• Protest activities (I02)– Two dimensional-structure (legal and
illegal protest)– Shortening of legal protest scale (two
items proposed for deletion)
NRCMeetingWindsor
June 2008
Perceptions of participation in society - 2
• Participation as an adult (I03)– Two dimensions (expected electoral
participation and expected active political participation)
– Good scaling properties for all items
• Participation as a young person (I04)– First two items reflect different dimension
(community-based participation)– Good scaling properties for item
measuring informal participation
NRCMeetingWindsor
June 2008
International option regarding religion
• Religious denomination (J01)
• Participation in religious services (J02)
• Attitudes toward religion (J03)– Good scaling properties for five items– Three items do not really reflect attitudes
and should be deleted
NRCMeetingWindsor
June 2008
Proposed student questionnaire for main survey
• With 5 items per minute the estimated length is 34 minutes
• There are about 170 to 180 items in the instrument
• The CIVED questionnaire had 162 items in the perceptions part (for 30 minutes)
NRCMeetingWindsor
June 2008
Proposed changes
• Reduction in questionnaire length of about 10 minutes
• Generally, only slight modifications or deletions are proposed
• The only more fundamental change for democratic value beliefs a format change has been dropped following PAC advice– Risk of using an un-trialled question format in
main survey!
NRCMeetingWindsor
June 2008
Questions or comments?
NRCMeetingWindsor
June 2008
Group discussions
• No complete re-writing of questions possible• No additional questions
– unless previously used in surveys of the same age group...
• Go through the original field trial material (tables in document 6) and discuss– Agreement with proposed selection– Prioritisation of questions – Feedback on proposed – Suggestions of (minor) changes to wording
NRCMeetingWindsor
June 2008
Group allocations
• From six groups
• Three groups start from the beginning of questionnaire material (those sitting in front)
• Three groups start from the end (those sitting towards the back)
• Try to form mixed groups (from different countries and regions)
