effects of design in web surveys
DESCRIPTION
Effects of Design in Web Surveys. Vera Toepoel Tilburg University The Netherlands. CentERdata: Two Online Panels. 1. CentERpanel: l Exists for 17 years 2000 households Respondents fill out questionnaires every week Online interviews as method, but: - PowerPoint PPT PresentationTRANSCRIPT
04/25/23
Effects of Design in Web Surveys
Vera ToepoelTilburg UniversityThe Netherlands
04/25/23
3
CentERdata: Two Online Panels1. CentERpanel:l
• Exists for 17 years• 2000 households• Respondents fill out questionnaires every
weekOnline interviews as method, but:
Probability sample drawn from address sampling frame of Statistics Netherlands
Recruitment of new panel members address-based
Includes households without internet access (less than 20%): Equipment
04/25/23
4
CentERdata: Two Online Panels2. LISS Panel• Grant from The Netherlands Organisation for
Scientific Research • 5000 households• Established in 2007 (we fielded 1st
questionnaire!)• Respondents fill out questionnaires every
monthOnline interviews as method, but:
Probability sample drawn from address sampling frame of Statistics Netherlands
Contacted by letter, telephone or visit Includes households without internet access (less
than 20%): Equipment
04/25/23
5
1 item per screen
04/25/23
6
4 items per screen
04/25/23
7
10 items per screen
04/25/23
8
04/25/23
9
Answer categories
04/25/23
10
Open-ended
04/25/23
11
Vertical: positive to negative
04/25/23
12
Horizontal
04/25/23
13
Numbers 1 to 5
04/25/23
14
Numbers 5 to 1
04/25/23
15
Numbers 2 to -2
04/25/23
16
Trained Respondents: Panel conditioning• Content (knowledge on topics)
– Prepare for future surveys– Develop attitudes
• Procedure (question-answering process)– Learn how to interpret questions– Answer strategically– Speed through the survey
04/25/23
17
Procedure (answer process)
• Differences between trained and fresh respondents with regard to web survey design choices– Items per screen– Response category effects– Question layout
04/25/23
18
Overall:
Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel.
04/25/23
19
Experiment 1: Items per screen
• Social Desirability Scale • 10 items• 3 different formats:
– 1 item per screen– 5 items per screen– 10 items per screen
04/25/23
20
Experiment 1: Items per screen
• Trained respondents had higher inter-item correlations for multiple-item-per-screen formats.
• No significant difference in item non-response.
• Mean score of the Social Desirability Scale showed no evidence for social desirability bias.
• The mean duration to complete the ten social desirability items did not differ significantly between panels.
04/25/23
21
Experiment 2: Answer Categories
04/25/23
22
Experiment 2: Answer Categories
04/25/23
23
Experiment 2: Answer Categories
• Category effect found• No difference in category effect between
trained and fresh respondents
04/25/23
24
Experiment 3: Question LayoutQuestion: Overall, how would you rate the
quality of education in the Netherlands?Answer: 5-point scale Six formats:1. Reference format (decremental)2. Reverse scale: incremental3. Horizontal layout4. Add numbers 1 to 5 to verbal labels5. Add numbers 5 to 1 to verbal labels6. Add numbers 2 to -2 to verbal labels
04/25/23
25
Experiment 3: Question Layout
1. Decremental vs. incremental: T+ F2. Vertical vs. horizontal layout: -3. No numbers vs. numbers 1 to 5:-4. Numbers 1 to 5 vs. numbers 5 to 1: T+F5. Numbers 5 to 1 vs. Numbers 2 to -2: T+F
• Trained respondents more easily selected one of the first options.
T=significant differences in Trained panelF=significant differences in Fresh panel
04/25/23
26
Design Effects in Web Surveys: Comparing Trained and Fresh Respondents
• Overall little differences between trained and fresh respondents
• Trained respondents are somewhat more sensitive to satisficing:– Shorter completion times– Higher inter-item correlations for multiple-
items-per-screen formats– Select first response options more often
04/25/23
27
Current and Future Research
• It has been little more than a decade since systematic research was begun on visual design effects in web surveys.
• In the last decade dozens of studies have been conducted
• It is now important that we begin to understand the importance of each of the visual effects
• Can we reduce visual effects by effective question writing?!
04/25/23
28
Effective Question Writing
Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to:
Verbal language>numbers> visual cues
Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers?
04/25/23
29
Experiment 1: Visual Heuristics(joint with Don Dillman)
• Tourangeau, Couper, and Conrad (POQ 2004; 2007):1. Middle means typical: respondents will see the middle
option as the most typical2. Left and top means first: the leftmost or top option
will be seen as the ‘first’ in conceptual sense3. Near means related: options that are physically near
each other are expected to be related conceptually4. Up means good: the top option will be seen as the
most desirable5. Like means close: visually similar options will be seen
as closer conceptually
Experimental conditions: • Polar point or fully labeled scale• With or without numbers (1 to 5)
04/25/23
30
Middle Means Typical
Fully labeled: even spacing
Fully labeled: uneven spacing
04/25/23
31
Left and Top Means First
Fully labeled with color: consistent ordering
Fully labeled with color: inconsistent ordering
04/25/23
32
Near Means Related
Polar point with numbers: separate screens
Polar point with numbers: single screen
04/25/23
33
Up Means Good
Polar point with numbers: incremental
Polar point with numbers: decremental
04/25/23
34
Like Means Close
Polar point
Polar point with color
04/25/23
35
Like Means Close
Polar point with numbers (1 to 5)
Polar point with different numbers (-2 to 2)
04/25/23
36
Labels, numbers and visual heuristics: is there a hierarchy?
1. Middle Means Typical
2.Left and Top Means First
3. Near Means Related
4. Up Means Good
5. Like Means Close
Effect heuristic?
no no yes yes yes
Numbers reduced effect?
yes yes Color: yesDif. #: no
Effect heuristic in fully labeled scales?
no no yes no no
04/25/23
37
Experiment 2: Pictures in web surveys (joint with Mick Couper)Replicate study Couper, Tourangeau, and
Kenyon (POQ 2004)– 1. No Picture– 2. Low frequency picture– 3. High frequency picture
Add verbal instructions– A. No verbal instruction– B. Instruction to include both high and low
frequency instances– C. Instruction to include only low
frequency instances
04/25/23
38
Low and High frequency picture
04/25/23
39
Can verbal instructions reduce the effects of pictures?MANOVA
– main effect instructions lambda=.597, p<.0001
– main effect pictures lambda=.964, p<.0001
– interaction instructions*pictures lambda=.9691,p<.0001
– This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures!
04/25/23
40
Future Research
How to reduce visual design effects in web surveys
04/25/23
41
LISS data
Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost Proposals can be submitted through
www.lissdata.nl Existing data free available for academic
use longitudinal core studies proposed studies disseminated through www.lissdata.nl