effects of mobile devices on data quality in web-based surveys · gor 2018 | effects of mobile...

GOR 2018 | 2nd March 2018

Effects of mobile devices on data quality in web-based surveys

2

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Device Ownership

90% 43% 58%

2017 2017 2016

Laufende Wirtschaftsrechnung, Statistisches Bundesamt

3


Mobile Devices

>15‘‘ 7-12‘‘ <6‘‘

External Keyboard

+ Mouse

Internal Keyboard

+ Touch

Internal Keyboard

+ Touch

Cable or WiFiWiFi or mobile

network

WiFi or mobile

network

4


Difficulties when Assessing the Effects

SELECTION EFFECTS

MEASUREMENT EFFECTS

5


Propensity Score Matching

AGE

GENDER

HOUSEHOLD INCOME

EDUCATION

EMPLOYMENT STATUS

EXPERIENCE WITH SURVEYS

AFFINITY FOR TECHNOLOGY

INTERNET USAGE

EXPERIENCE WITH DEVICE

Comparison of “similar” respondents within treatment and control group

QUASI EXPERIMENT

6


Data

Gesis Panel

10th wave (August 2015)

Not optimized for mobile

7


Indicators for Data Quality

LENGTH OF OPEN

ANSWERS

PRIMACY

EFFECTS

SURVEY

DURATION

8


Results

LENGTH OF OPEN ANSWERS

9


* p < 0,10 ** p < 0.05 *** p < 0.01

Results

- 23** Characters - 9 Characters + 3 Characters

LENGTH OF OPEN ANSWERS

10


Results

PRIMACY EFFECTS

11


+ 22 PP***more 1./2. answers

+ 1 PPmore 1./2. answers

+/- 0 PPmore 1./2. answers

* p < 0,10 ** p < 0.05 *** p < 0.01

Results

PRIMACY EFFECTS

12


Results

SURVEY DURATION

13


* p < 0,10 ** p < 0.05 *** p < 0.01

Results

+ 6.5** minutes + 4.6* minutes + 4.7** minutes

SURVEY DURATION

14


Lower Evaluation of the Study

Higher Item Nonresponse

Shorter Open Answers

Higher Primacy Effects

Stronger Left-Alignment

Longer Duration

Conclusion

15


Responsive

survey design

Item nonresponse:

Make answers

mandatory

Multiple choice

questions:

Randomize answer

options

Exclusion of

mobile devices or

asking the

respondents to

switch

Possible Solutions

Optimization for

and testing on a

variety of devices

16


Data

30 SKOPOS surveys

2017

Optimized for mobile devices

17


Participation by Device

Single survey

76%

24%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

18


Data Quality

Closed Questions

(Average std

in item set)

Open-Ended Questions

(Number of characters written)

0,860,96

60,5352,51

19


Completion Rate

Single survey

82%

77%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

20


When do respondents drop out? (Average seconds)

363 sec.333 sec.

21


But is survey length and drop-out rate correlated?

R² = 0,057

0%

25%

50%

75%

100%

0 5 10 15 20

Average interview duration (minutes)

Com

plet

ion

Rat

e R² = 0,0657

0%

25%

50%

75%

100%

0 5 10 15 20 25

Single survey

Average interview duration (minutes)

22


Conclusion

Survey

Duration

(Sweetspot: 5 min.)

Mobile ExperienceResponsive Design

23

GOR 2018 | Learning From All Answers | March 1, 2018

Further questions? Let‘s talk!

Thank You For Your Attention!

Patrick Schlickmann

Project Manager

[email protected]

Sebastian Schmidt

Director Research & Development

[email protected]

mailto:[email protected]

mailto:[email protected]

effects of mobile devices on data quality in web-based surveys · gor 2018 | effects of mobile...

Documents