assessment of bias when field operations are curtailed in a mixed-mode telephone and face-to-face...
Post on 01-Apr-2015
213 Views
Preview:
TRANSCRIPT
Assessment of Bias When Field Operations Are Curtailed in a Mixed-Mode Telephone and Face-
to-Face Survey of Persons with Disabilities
May 13, 2011
Presentation to the AAPOR Annual Conference
Eric Grau
Multi-mode designs are used to boost response rates while controlling costs
Single mode:– Greater potential for nonresponse bias
Multi-mode:– Greater potential for measurement error bias
May be more problematic in certain populations (e.g., persons with disabilities)
Introduction
2
Telephone (CATI) and face-to-face (CAPI) interviews are often used together in mixed- mode studies
Similar: both involve an interviewer
Different:– CAPI interviews allow for greater ability to develop
rapport and maintain the respondent’s interest– CAPI interviews tend to be easier to match pace and
communication style
Background
3
Literature:– Response errors in CATI are different than those that
occur in face-to-face interviews– Krosnick (2002): greater cognitive demand
more shortcuts by respondents– Respondents who require more effort to obtain an
interview provide poorer quality data, increasing measurement error
Background (continued)
4
CAPI is seen as necessary to adequately survey the population in studies of persons with disabilities
CATI is used to save money in studies with limited resources
Mixed-mode approaches (such as CATI/CAPI) can therefore be used in studies of persons with disabilities that have limited resources, where CAPI interviews are only used for some difficult-to-reach subpopulations
Background (continued)
5
This study: assume field operations reduce bias. Under this assumption: how much is bias reduced?
Is it possible to scale back or eliminate CAPI interviews without significant changes in the respondent population, or significant changes in outcomes?
Previous research: mode effects may not be trivial
Research Question
6
Ticket to Work (TTW) program provides participants with a coupon to obtain employment training and other employment-related services
Eligibility for TTW program:– Must be a beneficiary of one or both of two Social
Security Administration programs for persons with disabilities, ages 18-64:
• Supplemental Security Income (SSI)• Social Security Disability Insurance (SSDI)
Evaluation of the Ticket to Work (TTW) Program
7
Component surveys– Beneficiary Survey
• Target Population– All 10.4 million SSI/SSDI beneficiaries
– Participant Survey• Target Population
– Among 10.4 million SSI/SSDI beneficiaries, subpopulation of TTW participants (108,000)
Evaluation of the Ticket to Work (TTW) Program
8
Collects health and well-being, service use, and employment information from SSI and SSDI beneficiaries
Dual-mode survey (CATI/CAPI)
Length: 45 minutes
In 2006 (third round of survey), 6,600 respondents– 82% response rate– Approximately 5,100 by CATI and 1,500 by CAPI– 23% of respondents by CAPI interview
National Beneficiary Survey
9
Multiple-Stage Cluster Sample– For the beneficiary sample and most subpopulations
of the participant population
Sample Design
10
3,382 sample members– 1,945 resolved by phone (completes, ineligibles,
nonrespondents)– 1,437 fielded (42%)
2,508 completes– 1,645 completed by phone– 863 fielded (34% of completes)
• 741 fielded completes with 60 days of field effort• No fielded completes with no field efforts
Attributes of Beneficiary Sample: Number Fielded and Number Complete
11
Used paradata on respondents
Process details (in chronological order)– Codes describing each locating and/or interview
attempt– The date that entries were recorded– Notes about the locating or interview attempt
Current Study
12
For sample members that were assigned to field operations at some point, we reassigned a disposition code1. As if no field operations were conducted
2. As if field operations were limited to 60 days after the first field assignment
Once this was done, nonresponse adjustments were recalculated with new status codes
Disposition Code Reassignment
13
Number of Standard Errors
Respondent’s Disability Frame
5 Months of Field Effort
60 Days of Field Effort
No Field Effort
Deaf 0.9% 0.1 -0.4 -1.2
Blind 2.4% 1.6 1.6 1.6
Psychiatric Disability
30.4% -2.6 -2.7 -3.1
Intellectual Disability
13.5% 0.7 0.6 -0.6
Other Physical Disability
52.8% 1.3 1.5 2.9
Comparison with Frame: Representative Beneficiary Sample
14
Number of Standard Errors
Respondent’s Disability Frame
5 Months of Field Effort
60 Days of Field Effort
No Field Effort
Deaf 0.9% 0.1 0.3 -0.1
Blind 2.4% 1.0 1.2 1.0
Psychiatric Disability
30.4% -1.2 -1.1 -0.7
Intellectual Disability
13.5% 0.4 0.6 0.1
Other Physical Disability
52.8% 0.5 0.2 0.3
Comparison with Frame: Beneficiary Sample After NR Adjustments
15
Variables with Differences in– Item nonresponse
• More item nonresponse in CATI– Social desirability
• More likely to give socially desirable answers in CATI– Acquiescence
• CATI respondents “better informed”
Comparison of Survey Variables: Measures
16
Maximum Number of Standard Errors Away from Original Estimate
Survey Variable60 Days of Field Effort No Field Effort
Race 0.27 0.50
Father’s education 0.20 0.98
Mother’s education 0.38 0.72
Education level 0.36 0.56
General health 0.24 0.56
Health insurance 0.14 1.79
Comparison of Survey Variables: Representative Beneficiary Sample (Item NR Variables)
17
Maximum Number of Standard Errors Away from Original Estimate
Survey Variable60 Days of Field Effort No Field Effort
Goals include moving up 0.37 0.40
Used drugs in past year 0.20 0.01
Work for pay next year 0.44 0.90
Work for pay next 5 years 0.32 0.71
Household income (median)
0.03 1.12
Comparison of Survey Variables: Representative Beneficiary Sample (Social Desirability Variables)
18
Maximum Number of Standard Errors Away from Original Estimate
Survey Variable60 Days of Field Effort No Field Effort
Heard of impairment-related work expenses exclusion
0.29 1.39
Heard of expedited reinstatement
0.16 1.66
Heard of BPAO 0.36 2.36
Heard of TTW 0.28 1.01
Comparison of Survey Variables: Representative Beneficiary Sample (Acquiescence Variables)
19
Mode effects do exist for some key variables
Removing field effort is not recommended
Consistent with work by Sloan, Wright, and Barrett (2006)
Variables that are prone to “acquiescence” (that is, saying yes just to get through the interview) sensitive to reduced field effort
Conclusions
20
Reducing field efforts to 60 days did not have a major effect on estimates– Possible cost-saving measure
More research needed– Only three levels (no field effort, 60 days of field
effort, and 5 months of field effort)– Need to determine how much field effort is enough
to minimize bias
Conclusions (continued)
21
Eric Grauegrau@mathematica-mpr.com
Mathematica Policy Research600 Alexander ParkPrinceton, NJ 08540
www.Mathematica-MPR.com
Contact Information
22
top related