investigating the effects of questionnaire design and ... · pdf file98 2000 2002 2004 2006...
TRANSCRIPT
www.som.gu.se
Elias Markstedt, Frida VernersdotterThe SOM Institute,University of Gothenburg, Sweden
Prepared for AAPOR 2013
Investigating the Effects ofQuestionnaire Design andQuestion Characteristics onRespondent Fatigue
www.som.gu.se
Research Question
• What questionnaire design and respondentcharacteristics are crucial to explain breakoff patterns?
• Why? Because item nonresponse due to breakoffsdiffer from other kinds of item nonresponse
• A replication of Peytchev (2009), a study on web-survey breakoffs
www.som.gu.se
The SOM Data
SOM Institute Dataset• Mail-back survey, random sample of the Swedish adult
population, ages 15-85• A wide range of questions - political attitudes, media
use and lifestyle etc.• We use data from 2000-2012, 31 questionnaires, 641
pages• Over 52,000 respondents
(AAPOR RR5 49%-57%)
www.som.gu.se
Data – Unit Response Rates
6259
57 58 5755
63 62
5658
55 5558
56
5255 55
5755 55
5153 52 51 51
49 49
6870
6966 66 67
71 7067 65
69 69 6967
6367
65 66 6563
6063
58 59 6057 57
0
10
20
30
40
50
60
70
80
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
AAPOR RR5
AAPOR RR6
SOM-standard Brutto
SOM-standard Netto
Svarsfrekvens (%)
6259
57 58 5755
63 62
5658
55 5558
56
5255 55
5755 55
5153 52 51 51
49 49
6870
6966 66 67
71 7067 65
69 69 6967
6367
65 66 6563
6063
58 59 6057 57
0
10
20
30
40
50
60
70
80
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
AAPOR RR5
AAPOR RR6
SOM Standard Gross RR
SOM Standard Net RR
Response Rate (%)
www.som.gu.se
Data – Unit Response Rates
6259
57 58 5755
63 62
5658
55 5558
56
5255 55
5755 55
5153 52 51 51
49 49
6870
6966 66 67
71 7067 65
69 69 6967
6367
65 66 6563
6063
58 59 6057 57
0
10
20
30
40
50
60
70
80
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
AAPOR RR5
AAPOR RR6
SOM-standard Brutto
SOM-standard Netto
Svarsfrekvens (%)
6259
57 58 5755
63 62
5658
55 5558
56
5255 55
5755 55
5153 52 51 51
49 49
6870
6966 66 67
71 7067 65
69 69 6967
6367
65 66 6563
6063
58 59 6057 57
0
10
20
30
40
50
60
70
80
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
AAPOR RR5
AAPOR RR6
SOM Standard Gross RR
SOM Standard Net RR
Response Rate (%) 2000 onward
www.som.gu.se
Measuring Breakoff
• Who do we measure?• There are few respondents who make clean breaks and
send in their questionnaires (≈1%) – “true” breakoffs• “Temporary” or “Partial” breakoffs: sequence of
item nonresponses > 30 items missing in a row• “True” + “Temporary” breakoff rate ≈ 4.5%• What triggers that sequence?
www.som.gu.se
What do we measure?
www.som.gu.se
What do we measure?
”True” breakoffs
www.som.gu.se
What do we measure?
”Temporary”breakoffs
www.som.gu.se
Explanatory Variables –Questionnaire Design
• Number of questions/page• Cumulative number of questions• Open-ended questions• Long questions (>119 characters)• Sensitive questions• Judgment questions• Definitions (fully labeled scale)• Total no of items in question battery• “No opinion” option
www.som.gu.se
Explanatory Variables –Respondent Characteristics
• Age• Gender• High education• High income• Non-working (pensioners, students, unemployed etc.)• Political interest
www.som.gu.se
Results: Hazard Survival ModelPredictor
Relative risk(Odds ratios) p-value
QuestionnaireDesign: Number of items/page 1.06 0.000
Cumulative number of questions 0.99 0.000QuestionCharacteristics: Total no. of items in question battery 0.93 0.000
Open-ended questions 1.83 0.000Long questions 1.31 0.000Sensitive questions 1.46 0.000Judgment questions 0.90 0.004Fully labeled scale 0.65 0.000”No opinion” option 1.28 0.000
RespondentCharacteristics: Non-working 0.80 0.000
High education 0.74 0.000High income 0.67 0.000Age 1.01 0.000Gender (1=female) 1.06 0.065Political interest 0.57 0.000
Numbers at risk/number of breakoffs: 52,551/3,930
www.som.gu.se
Results: Hazard Survival ModelPredictor
Relative risk(Odds ratios) p-value
QuestionnaireDesign: Number of items/page 1.06 0.000
Cumulative number of questions 0.99 0.000QuestionCharacteristics: Total no. of items in question battery 0.93 0.000
Open-ended questions 1.83 0.000Long questions 1.31 0.000Sensitive questions 1.46 0.000Judgment questions 0.90 0.004Fully labeled scale 0.65 0.000”No opinion” option 1.28 0.000
RespondentCharacteristics: Non-working 0.80 0.000
High education 0.74 0.000High income 0.67 0.000Age 1.01 0.000Gender (1=female) 1.06 0.065Political interest 0.57 0.000
Numbers at risk/number of breakoffs: 52,551/3,930
www.som.gu.se
Results: Hazard Survival ModelPredictor
Relative risk(Odds ratios) p-value
QuestionnaireDesign: Number of items/page 1.06 0.000
Cumulative number of questions 0.99 0.000QuestionCharacteristics: Total no. of items in question battery 0.93 0.000
Open-ended questions 1.83 0.000Long questions 1.31 0.000Sensitive questions 1.46 0.000Judgment questions 0.90 0.004Fully labeled scale 0.65 0.000”No opinion” option 1.28 0.000
RespondentCharacteristics: Non-working 0.80 0.000
High education 0.74 0.000High income 0.67 0.000Age 1.01 0.000Gender (1=female) 1.06 0.065Political interest 0.57 0.000
Numbers at risk/number of breakoffs: 52,551/3,930
www.som.gu.se
Results: Hazard Survival ModelPredictor
Relative risk(Odds ratios) p-value
QuestionnaireDesign: Number of items/page 1.06 0.000
Cumulative number of questions 0.99 0.000QuestionCharacteristics: Total no. of items in question battery 0.93 0.000
Open-ended questions 1.83 0.000Long questions 1.31 0.000Sensitive questions 1.46 0.000Judgment questions 0.90 0.004Fully labeled scale 0.65 0.000”No opinion” option 1.28 0.000
RespondentCharacteristics: Non-working 0.80 0.000
High education 0.74 0.000High income 0.67 0.000Age 1.01 0.000Gender (1=female) 1.06 0.065Political interest 0.57 0.000
Numbers at risk/number of breakoffs: 52,551/3,930
www.som.gu.se
Results: Hazard Survival Model –Compared with Peytchev’s results
PredictorRelative risk –Our model
Relative risk –Peytchev
Questionnaire Design: Number of questions/page ↑ ↑Cumulative number of questions ≈0 ≈0
QuestionCharacteristics:
Total no. of items in questionbattery ↓ .Open-ended questions (1=yes) ↑ ↑Long questions (1=yes) ↑ ↑Sensitive questions ↑ ↓Judgment questions ↓ ↑Fully labeled scales ↓ ↑”No opinion” option ↑ .
RespondentCharacteristics: Non-working ↓ ↓
High education ↓ ↓High income ↓ ↓Age ≈0 ≈0Gender (1=female) ↑ ↓Political interest ↓ .
www.som.gu.se
Conclusions
• We replicated most of the findings in Peytchev (2009)• Look for both “true” and “temporary” breakoffs• Reducing breakoffs:
– Less demanding questions– Open questions– Sensitive questions– Shorter questions– Fully labeled scales
www.som.gu.se
Thank you!
Contact information:Elias Markstedt
The SOM Institute
University of Gothenburg, Sweden
www.som.gu.se
Results: Hazard Survival Model –Compared with Peytchev’s results
PredictorRelative risk –Our model p
Relative risk –Peytchev’s model p
QuestionnaireDesign: Number of questions/page 1.06 0.000 1.23 <0.001
Cumulative number of questions 0.99 0.000 1.01 0.419QuestionCharacteristics:
Total no. of items in questionbattery 0.93 0.000 -Open-ended questions (1=yes) 1.83 0.000 2.39 .029Long questions (1=yes) 1.31 0.000 2.90 .034Sensitive questions 1.46 0.000 0.64 .239Judgment questions 0.90 0.004 1.46 .189Fully labeled scales 0.65 0.000 1.10 .773”No opinion” option 1.28 0.000 -
RespondentCharacteristics: Non-working 0.80 0.000 0.84 .236
High education 0.74 0.000 0.89/0.57 .244/.174High income 0.67 0.000 0.79/0.62 .271/.051Age 1.01 0.000 0.99 .174Gender (1=female) 1.06 0.065 1.12 (male)Political interest 0.57 0.000 -