the dual tasks of interviewers ting yan colm o‘muircheartaigh jenny kelly pat cagney rebecca...
TRANSCRIPT
![Page 1: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/1.jpg)
The Dual Tasks of Interviewers
Ting YanColm O‘Muircheartaigh
Jenny KellyPat Cagney
Rebecca JessoeNORC at University of Chicago
Kenneth RasinskiUniversity of Chicago
Gary EulerCenters for Disease Control and Prevention
![Page 2: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/2.jpg)
What do interviewers do?
• Recruiting potential respondents– Introducing survey to potential respondents– Gaining cooperation – Screening for eligible respondents
• Administering interviews– Reading questions– Recording answers– Probing– Providing definitions
![Page 3: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/3.jpg)
Desired qualities of interviewers
• When recruiting respondents– Adaptive and flexible (Converse & Schuman, 1974)
• Tailoring (Groves & McGonagle, 2001; Houtkoop-Steenstra & van den Bergh, 2002; Maynard & Schaeffer, 2002)
• Maintaining interaction (Groves & McGonagle, 2001)• Those who developed their own approach had lower refusal
and higher cooperation than those who follow a standard script
• When administering interviews– Technician like (Converse & Schuman, 1974)
• Standardized interviewing (Fowler and Magione, 1990)
• Conflicting?
![Page 4: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/4.jpg)
How do interviewers affect survey error?
• Recruiting respondents– Nonresponse error
• If interviewer consistently attract respondents with a certain characteristic
• Administering interviews– Measurement error
• Interviewer bias • Interviewer variance
– If interviewers consistently influence responses in a certain way
![Page 5: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/5.jpg)
Research questions
• Is there a relationship between interviewers’ performance at recruiting respondents and administering interviews? – Are interviewers who are good at recruiting
respondents also good at collecting data of good quality?
• How does interviewer experience mediate this relationship, if the relationship exists?
![Page 6: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/6.jpg)
Data
• National Immunization Survey (NIS)– Nationwide, list-assisted random digit-dialing
(RDD) survey conducted by the NORC for the Centers for Disease Control and Prevention
– Monitors the vaccination rates of children between the ages of 19 and 35 months.
– 2007 Q3 data• 712 interviewers worked • 499,490 telephone numbers dialed
– 4,438 interviews obtained
![Page 7: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/7.jpg)
Which interviewers were includedin the analysis?
• Interviewers who had completed interview(s) on first contact– 295 interviewers
• 3114 completes
![Page 8: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/8.jpg)
Measures of recruitment task
• (First contact) Refusal rate =# refusals/# first contact cases
• (First contact) Completion rate=# completes/# first contact cases
• (First contact) Eligibility rate=# eligibles/# first contact cases
• Denominator: first contact cases– Virgin (fresh) cases or cases that were dialed by
autodialers only. – They haven’t been touched by humans before sent to the
current interviewer.
• Refusal conversion rate=# converted refusals/# refusals
![Page 9: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/9.jpg)
Measures of administration task
• Interviewer effect (ρint)
• Adherence to standardized interviewing (monitoring data)
• Item nonresponse
• Interview time (cost)
![Page 10: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/10.jpg)
Good openers vs. Bad openers
• Good openers: 3 out of 4 rates are above medians
Good Openers
Bad Openers
# of interviewers 100 195
Average # of interviews 13 9
Refusal Rate 10.46% 13.09%
Refusal Conversion Rate 1.34% 0.32%
Completion Rate 0.30% 0.15%
Eligibility Rate 3.91% 2.37%
![Page 11: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/11.jpg)
Good openers vs. Bad Openers (II)
• When experience is introduced– Median split on # of days worked at NORC
Number of
InterviewersAverage Number of
Interviews Completed
Good
OpenersBad
OpenersGood
OpenersBad
Openers
Experienced 67 83 15 11
Inexperienced 33 112 11 8
![Page 12: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/12.jpg)
Good openers vs. Bad Openers (III)
• When experience is introduced– Median split on # of days worked at NORC
Refusal
Rate
Refusal Conversion
RateCompletion
RateEligibility
Rate
Good Bad Good Bad Good Bad Good Bad
Experienced 10.5% 12.5% 1.7% 0.6% 0.32% 0.16% 4.1% 2.2%Inexperienced 10.5% 13.5% 0.5% 0.1% 0.27% 0.15% 3.6% 2.5%
![Page 13: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/13.jpg)
ρint
• ρint : Intra-interviewer correlation• Deffint=1+ ρint*(m-1)• Hierarchical linear models
– Respondent data as level 1 data– Interviewer data as level 2 data– Unconditional model with no explanatory
variables at either level– ρint=between-interviewer variance/total
variance
![Page 14: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/14.jpg)
ρint (II)
Family Income
Good openers Bad openers
Experienced 0.0825 0.0337
Inexperienced 0.2395 0.1053
0.1236 0.0786
![Page 15: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/15.jpg)
ρint (III)
# of people living in household
Good openers Bad openers
Experienced 0.0082 0.0408
Inexperienced 0.0004 0.0139
0.0013 0.0239
![Page 16: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/16.jpg)
ρint (IV)
# of Vaccines Received (Average)
Good openers Bad openers
Experienced 0.0003 0.0003
Inexperienced 0.0041 0.0026
0.0234 0.0085
![Page 17: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/17.jpg)
Monitoring scores
• Monitoring items– Read questionnaire verbatim– Probe without biasing or leading/Probing for Don’t
Knows– Reads scales as directed etc.
• Scores– 1=Error– 2=No Error– 3=Outstanding
• Item-level monitor score for each interviewer• Overall summary score for each interviewer
![Page 18: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/18.jpg)
Monitoring scores (II)
• Good openers on average have higher mean scores than bad openers, but difference sig. only for one monitoring item– Read Questionnaire Verbatim– Verifies dates and confirms spelling– Properly obtains all provider information– Use job aids as needed– Reads scales as directed– Records open-end response verbatim– Probes without biasing or leading/Probes Don’t
Knows
![Page 19: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/19.jpg)
Summary scores across monitoring items
6.80
7.00
7.20
7.40
7.60
7.80
8.00
8.20
8.40
8.60
Good Opener Bad Opener
Experienced
Inexperienced
![Page 20: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/20.jpg)
Item Nonresponse• A set of 24 questions every one had to answer• Item nonresponse rate=# of times R didn’t
provide an answer /24
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
G ood Opener B ad Opener
E xperienc edInexperienc ed
![Page 21: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/21.jpg)
Average Interview Duration (cost)
• Time spent on completing an interview– The longer the interview time, the more costly
27
28
29
30
31
32
33
34
G ood Opener B ad Opener
E xperienc edInexperienc ed
![Page 22: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/22.jpg)
Provider consent rate
66%
68%
70%
72%
74%
76%
78%
80%
82%
Good Opener Bad Opener
Experienced
Inexperienced
(79.8%) (74.9%)
![Page 23: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/23.jpg)
Conclusions and Discussion
• Good-opener interviewers • More completes• Higher refusal conversion, completion, and eligibility rates• Lower refusal rate
• Good-opener interviewers• Higher intra-interviewer correlation• But more adherence to standardized interviews (higher
monitoring scores)• More missing data
• Are good openers also good at collecting data of good quality?
• No one clear answer• Depends on which measures of interviewing tasks
• Experience didn’t matter much
![Page 24: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/24.jpg)
Limitations and Next Steps
• Only used various rates to measure interviewers’ performance at the recruitment stage
• Demographic compositions by interviewer status• Nonresponse error by interviewer status
• Only used proxy measures of data quality• Direct measures of measurement bias
• Interviewer characteristics and respondent characteristics not considered
• Bringing in interviewer and respondents characteristics into the picture
• Examining the effect of matched interviewer and respondents characteristics
![Page 26: The Dual Tasks of Interviewers Ting Yan Colm O‘Muircheartaigh Jenny Kelly Pat Cagney Rebecca Jessoe NORC at University of Chicago Kenneth Rasinski University](https://reader030.vdocuments.site/reader030/viewer/2022032703/56649d005503460f949d1a62/html5/thumbnails/26.jpg)