effects of survey mode on responses: an experiment of social desirability biases in telephone,...
TRANSCRIPT
![Page 1: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/1.jpg)
Effects of Survey Mode on Responses:
An Experiment of Social Desirability Biases in Tele-phone, Interviewer, and Online Surveys
Indeok Song
Joongbu University, Korea
Sung Kyum Cho
Chungnam National University
![Page 2: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/2.jpg)
Contents
• Background: 3 Surveys on Gambling Addiction
• Research Question & Purpose
• Review of Previous Research
• Methods
• Results
• Discussions & Suggestions
![Page 3: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/3.jpg)
Results of 2010’s Surveys
Online (n=581) Telephone (n=1,797)
Interview (n=1,000)0
0.51
1.52
2.53
3.54
3.83
0.32 0.5
Means on Gambling Addiction Scale (CPGI 0~27)
F (2, 3,375) = 402.72, p < .001
![Page 4: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/4.jpg)
Results of 2010’s Surveys
Online (n=581)
Telephone (n=1,797)
Interview (n=1,000)
0%10%20%30%40%50%60%70%80%90%
100%
35.6
95.2 87.123.1
1.6 6.8
19.1
1.4 4.422.2
1.8 1.7
ProblemModerate RiskLow RiskNon-Problem
χ²= 1,130.35, df = 6, N=3,378, p < .001
![Page 5: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/5.jpg)
Research Question What is the reason for the inconsistencies
among 3 surveys’ results? Which mode does produce the most accurate estimates?
Sampling? Survey Mode?
![Page 6: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/6.jpg)
Previous Research
• Survey modes “can be equivalent but are not always identical” (Riva et al., 2003, p. 79)
–Sensitivity & Social desirability (Booth-Kewley, Lar-
son, & Miyoshi, 2007; Couper, 2000; Huang, 2006; Joinson, 1999; Kays, Gather-coal, & Buhrow, 2011; Tourangeau & Yan, 2007, 2011 and so on)
• Mode of admin.: Self- vs. Interviewer-administered– Self-administration increases reporting of socially un-
desirable behaviors (Corkrey & Parkinson, 2002; Currivan et al., 2004; Hochstim, 1967; Moskowitz, 2004; Mott, 1985; Richman et al., 1999, and so on)
∴ Paper-Pencil, Online > Telephone, Interviewer
![Page 7: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/7.jpg)
Previous Research
• Interviewer or 3rd party presence– Social desirability bias is worse in telephone sur-
vey than in face-to-face interviews (Aquilino, 1994; Groves & Kahn, 1979; Holbrook et al., 2003; Johnson, Hougland, & Clay-ton, 1989; Leeuw & van der Zouwen, 1988, and so on)
– The presence of others affects responses to sensi-tive questions (Aquilino, Wright, & Supple, 2000; Harrison, 2001; Lombard & Ditton, 1997; Sudman, 2001; Tourangeau & Yan, 201, and so
on)
– Online survey yields the lowest level of other peo-ple’s presence
∴ Online > Interviewer > Telephone
![Page 8: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/8.jpg)
Research Purpose
Controlling for variances from sampling and dif-ferent groups of samples,
to examine the effects of survey modes on the responses to sensitive questions.
![Page 9: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/9.jpg)
Methods
• Pretest-posttest 3 between-group experi-mental design
– 93 undergraduate students 66 analyzed• Out of 66, male 38 (57.6%), female 28 (42.4%),
mean age 20.61 (SD=1.97)
Paper-pencil(N=93) 2 weeks Random
Assignment
Online(n=21)
Telephone(n=24)
Interview(n=21)
![Page 10: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/10.jpg)
Methods
• Questionnaire Design– Addiction Scales
• Gambling addiction (4pt., 9 items) – M=9.24, SD=2.54, α=.84
• Internet addiction (4pt., 15 items)– M=29.13, SD=7.24, α=.90
• Game addiction (4pt., 20 items)– M=26.80, SD=8.84, α=.96
– Dissatisfaction with self-appearance (5pt., 6 items)
• M=14.43, SD=3.71, α=.66
– Sexual values (5pt., 10 items)• M=27.88, SD=7.14, α=.89
![Page 11: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/11.jpg)
Methods
– Sexual experience• Age of the first experience
– n=27, M=18.59, SD=1.67
• Number of sexual partners– n=26, M=10.81, SD=3.68
• Sexual behaviors (physical contact, cuddle, kiss, caress, intercourse) experienced in current rela-tionship
– n=19, M=3.68, SD=1.42
– Demographics• Gender, age, household-income
![Page 12: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/12.jpg)
Results
• Homogeneity among groups (pretest: paper-pen-cil)
Source df F p
Gambling Addiction (2, 63) 1.29 .28
Internet Addiction (2, 63) .09 .91
Game Addiction (2, 63) 2.13 .13
Dissatisfaction with Self (2, 63) .31 .73
Sexual Values (2, 63) .28 .76
Age of 1st Experience (2, 24) 1.89 .19
No. of Partners (2, 23) .59 .56
Sexual Behaviors (2, 16) .24 .79
![Page 13: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/13.jpg)
Responses for Game Addiction
Paper-Pencil Posttest20
22
24
26
28
30
32
28.24
30.71
24.5
21.92
28
24.91OnlineTelephoneInterview
Time * Mode: F (2, 63) = 3.35, p = .041, eta² = .10
![Page 14: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/14.jpg)
Responses for Dissatisfaction with Self-Appearance
Paper-Pencil Posttest12
12.5
13
13.5
14
14.5
15
13.95
14.33
14.83
12.29
14.38
13.19OnlineTelephoneInterview
Time * Mode: F (2, 63) = 5.58, p = .006, eta² = .15
![Page 15: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/15.jpg)
2 x 3 Factorial ANOVAs• Independent Variables
– 2 groups (‘Low’ & ‘High’) by median split– 3 modes (Online, Telephone, Interview)
• Dependent Variables– Changes in responses = Pretest – Posttest
DVs N M SD
Gambling Addiction 66 .27 2.30
Internet Addiction 66 1.08 6.78
Game Addiction 66 1.14 8.05
Dissatisfaction with Self 66 1.18 3.13
Sexual Values 66 1.44 4.65
Age of 1st Experience 22 .68 1.49
No. of Partners 22 -3.95 16.76
Sexual Behaviors 16 -.13 1.46
![Page 16: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/16.jpg)
Changes in Responses for Gambling Addiction
Online Telephone Interview
-2-1.5
-1-0.5
00.5
11.5
22.5
3
-0.080 -0.07
1.22
2.5
-1.83
Low GroupHigh Group
Mode: F (2, 60) = 5.32, p = .007, eta² = .15G * M: F (2, 60) = 5.06, p = .009, eta² = .14
![Page 17: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/17.jpg)
Changes in Responses for Internet Addiction
Online Telephone Interview
-5
-3
-1
1
3
5
7
-3-4.17
0.5
1.82
5.586.44
Low GroupHigh Group
Group: F (1, 60) = 23.15, p < .001, eta² = .28
![Page 18: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/18.jpg)
Changes in Responses forGame Addiction
Online Telephone Interview
-10-8-6-4-202468
10
-7.56
-0.2 -0.441.33
7.225.75
Low GroupHigh Group
Group: F (1, 60) = 18.54, p < .001, eta² = .24Mode: F (2, 60) = 5.65, p = .006, eta² = .16
![Page 19: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/19.jpg)
Changes in Responses for Dissatis-faction with Self-Appearance
Online Telephone Interview
-2
-1
0
1
2
3
4
-1.75
1.361.08
1.44
3.54
1.33
Low GroupHigh Group
Group: F (1, 60) = 7.41, p = .008, eta² = .11Mode: F (2, 60) = 4.89, p = .011, eta² = .14
![Page 20: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/20.jpg)
Changes in Responses for Sexual Values
Online Telephone Interview-1
0
1
2
3
4
5
6
7
0 0.29 0
1.6
5.8
1.7Low GroupHigh Group
Group: F (1, 60) = 7.32, p = .009, eta² = .11
![Page 21: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/21.jpg)
Changes in Responses for Age of the 1st Intercourse
Online Telephone Interview-1
0
1
2
3
4
5
6
7
0.330.67
0.29
6
0.5
Low GroupHigh Group
Group: F (1, 17) = 16.90, p = .001, eta² = .50Mode: F (2, 17) = 7.56, p = .004, eta² = .47G * M: F (2, 17) = 19.01, p < .001, eta² = .53
![Page 22: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/22.jpg)
Summary of the Results
• Responses for ‘Game Addiction’ & ‘Dissatisfaction with Self-Appearance’– Increased in the online survey mode– Decreased in the telephone & interview survey modes
• Those who scored higher on the sensitive questions are more likely to change their responses in a so-cially desirable direction– Less addicted (gambling, Internet, game)– Less dissatisfied with self-appearance– Less liberal (more conservative) sexual values– Younger in the age of the 1st sexual intercourse
![Page 23: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/23.jpg)
Discussions & Suggestions• Survey responses to sensitive (privacy-related) questions are
affected by different survey modes.– Online survey in which respondents are less concerned
about their privacies is more likely to elicit franker (more accurate) responses.
• Paper-pencil (group administration), telephone, interview modes lead respondents to privacy concern & misreport.
• Suggestion: Dual-mode surveyTelephone Sampling Online Questionnaire
![Page 24: Effects of Survey Mode on Responses: An Experiment of Social Desirability Biases in Telephone, Interviewer, and Online Surveys Indeok Song Joongbu University,](https://reader035.vdocuments.site/reader035/viewer/2022062404/5517094355034603568b5262/html5/thumbnails/24.jpg)
Any idea or sug-gestion
would be wel-
comed.