effects of design in web surveys

04/25/23

Effects of Design in Web Surveys

Vera ToepoelTilburg UniversityThe Netherlands

04/25/23

3

CentERdata: Two Online Panels1. CentERpanel:l

• Exists for 17 years• 2000 households• Respondents fill out questionnaires every

weekOnline interviews as method, but:

Probability sample drawn from address sampling frame of Statistics Netherlands

Recruitment of new panel members address-based

Includes households without internet access (less than 20%): Equipment

04/25/23

4

CentERdata: Two Online Panels2. LISS Panel• Grant from The Netherlands Organisation for

Scientific Research • 5000 households• Established in 2007 (we fielded 1st

questionnaire!)• Respondents fill out questionnaires every

monthOnline interviews as method, but:

Probability sample drawn from address sampling frame of Statistics Netherlands

Contacted by letter, telephone or visit Includes households without internet access (less

than 20%): Equipment

04/25/23

5

1 item per screen

04/25/23

6

4 items per screen

04/25/23

7

10 items per screen

04/25/23

8

04/25/23

9

Answer categories

04/25/23

10

Open-ended

04/25/23

11

Vertical: positive to negative

04/25/23

12

Horizontal

04/25/23

13

Numbers 1 to 5

04/25/23

14

Numbers 5 to 1

04/25/23

15

Numbers 2 to -2

04/25/23

16

Trained Respondents: Panel conditioning• Content (knowledge on topics)

– Prepare for future surveys– Develop attitudes

• Procedure (question-answering process)– Learn how to interpret questions– Answer strategically– Speed through the survey

04/25/23

17

Procedure (answer process)

• Differences between trained and fresh respondents with regard to web survey design choices– Items per screen– Response category effects– Question layout

04/25/23

18

Overall:

Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel.

04/25/23

19

Experiment 1: Items per screen

• Social Desirability Scale • 10 items• 3 different formats:

– 1 item per screen– 5 items per screen– 10 items per screen

04/25/23

20

Experiment 1: Items per screen

• Trained respondents had higher inter-item correlations for multiple-item-per-screen formats.

• No significant difference in item non-response.

• Mean score of the Social Desirability Scale showed no evidence for social desirability bias.

• The mean duration to complete the ten social desirability items did not differ significantly between panels.

04/25/23

21

Experiment 2: Answer Categories

04/25/23

22

Experiment 2: Answer Categories

04/25/23

23

Experiment 2: Answer Categories

• Category effect found• No difference in category effect between

trained and fresh respondents

04/25/23

24

Experiment 3: Question LayoutQuestion: Overall, how would you rate the

quality of education in the Netherlands?Answer: 5-point scale Six formats:1. Reference format (decremental)2. Reverse scale: incremental3. Horizontal layout4. Add numbers 1 to 5 to verbal labels5. Add numbers 5 to 1 to verbal labels6. Add numbers 2 to -2 to verbal labels

04/25/23

25

Experiment 3: Question Layout

1. Decremental vs. incremental: T+ F2. Vertical vs. horizontal layout: -3. No numbers vs. numbers 1 to 5:-4. Numbers 1 to 5 vs. numbers 5 to 1: T+F5. Numbers 5 to 1 vs. Numbers 2 to -2: T+F

• Trained respondents more easily selected one of the first options.

T=significant differences in Trained panelF=significant differences in Fresh panel

04/25/23

26

Design Effects in Web Surveys: Comparing Trained and Fresh Respondents

• Overall little differences between trained and fresh respondents

• Trained respondents are somewhat more sensitive to satisficing:– Shorter completion times– Higher inter-item correlations for multiple-

items-per-screen formats– Select first response options more often

04/25/23

27

Current and Future Research

• It has been little more than a decade since systematic research was begun on visual design effects in web surveys.

• In the last decade dozens of studies have been conducted

• It is now important that we begin to understand the importance of each of the visual effects

• Can we reduce visual effects by effective question writing?!

04/25/23

28

Effective Question Writing

Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to:

Verbal language>numbers> visual cues

Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers?

04/25/23

29

Experiment 1: Visual Heuristics(joint with Don Dillman)

• Tourangeau, Couper, and Conrad (POQ 2004; 2007):1. Middle means typical: respondents will see the middle

option as the most typical2. Left and top means first: the leftmost or top option

will be seen as the ‘first’ in conceptual sense3. Near means related: options that are physically near

each other are expected to be related conceptually4. Up means good: the top option will be seen as the

most desirable5. Like means close: visually similar options will be seen

as closer conceptually

Experimental conditions: • Polar point or fully labeled scale• With or without numbers (1 to 5)

04/25/23

30

Middle Means Typical

Fully labeled: even spacing

Fully labeled: uneven spacing

04/25/23

31

Left and Top Means First

Fully labeled with color: consistent ordering

Fully labeled with color: inconsistent ordering

04/25/23

32

Near Means Related

Polar point with numbers: separate screens

Polar point with numbers: single screen

04/25/23

33

Up Means Good

Polar point with numbers: incremental

Polar point with numbers: decremental

04/25/23

34

Like Means Close

Polar point

Polar point with color

04/25/23

35

Like Means Close

Polar point with numbers (1 to 5)

Polar point with different numbers (-2 to 2)

04/25/23

36

Labels, numbers and visual heuristics: is there a hierarchy?

1. Middle Means Typical

2.Left and Top Means First

3. Near Means Related

4. Up Means Good

5. Like Means Close

Effect heuristic?

no no yes yes yes

Numbers reduced effect?

yes yes Color: yesDif. #: no

Effect heuristic in fully labeled scales?

no no yes no no

04/25/23

37

Experiment 2: Pictures in web surveys (joint with Mick Couper)Replicate study Couper, Tourangeau, and

Kenyon (POQ 2004)– 1. No Picture– 2. Low frequency picture– 3. High frequency picture

Add verbal instructions– A. No verbal instruction– B. Instruction to include both high and low

frequency instances– C. Instruction to include only low

frequency instances

04/25/23

38

Low and High frequency picture

04/25/23

39

Can verbal instructions reduce the effects of pictures?MANOVA

– main effect instructions lambda=.597, p<.0001 

– main effect pictures lambda=.964, p<.0001

– interaction instructions*pictures lambda=.9691,p<.0001

– This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures!

04/25/23

40

Future Research

How to reduce visual design effects in web surveys

04/25/23

41

LISS data

Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost Proposals can be submitted through

www.lissdata.nl Existing data free available for academic

use longitudinal core studies proposed studies disseminated through www.lissdata.nl