redirected inbound call sampling (rics)– an example of fit ... · rics is a potential solution...
TRANSCRIPT
www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
Redirected Inbound Call Sampling (RICS)–An Example of Fit for Purpose Non-probability
Sample Design Burton Levine and Karol Krotki
FCSM Research and Policy ConferenceMarch 7, 2018
Need and ProblemNeed: “Federal statistical agencies must be in a position to provide objective, accurate, and timely information that is relevant to issues of public policy.”
Problem:“Continued reliance on (probability*) sample surveys as the principal means of collecting national statistical data is threatened by the increasing difficulty and cost in conducting the surveys, with consequent threats to data quality, and by the increasing demand for more and faster information.”*added by the presenter
-Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy (2017)
RICS is a Potential Solution
Redirected Inbound Call Sampling (RICS)
§ Low cost– About one-tenth the cost compared to outbound
telephone sampling § Quick
– Can collect data on thousands of respondents a day§ Low burden
– The respondent is already on the phone
Introducing Redirected Inbound Call Sampling (RICS) Surveys
RICS survey participants come from:§ Calls to nonworking toll-free numbers (e.g., Area Code:
800, 888)§ Calls to direct inward dialing (toll) calls that fail to connect
to their intendent target
RICS Data Collection Methods
VoiceKeypad
RICS Data Collection Methods
Interactive Voice
ResponseWeb Live
Interviewer
Redirected Inbound Toll-free Calls are Commonplace
Redirected inbound call sampling is commonplace§ 9% of toll-free number are
redirected § 5.4 million toll-free telephone
numbers are redirected.
Area code
n Redirected(%)
800 126 29.4844 126 0.8855 126 0.0866 126 7.9877 126 7.1888 126 8.7Total 756 9.0 (7.0, 11.0)
Innovation is applying RICS to population surveillance.
RICS Surveys Fielded by Presenter
Date Survey Funding organization Respondents
September 2015 BRFSS Evaluation Study RTI 6,799October 2016 National Adult Tobacco Survey NYS-DOH 4,302March 2017 NHIS Evaluation Study RTI 9,478April 2017 New York City Sleep Study NYC-DHMH 1,532August 2017 National Adult Tobacco Survey NYS-DOH 4,630In-progress IVR Evaluation Study RTI TBDIn-progress National Adult Tobacco Study NYS-DOH 4,000
NHIS Evaluation Study
We created a 27-question instrument that mimics questions from the National Health Interview Survey (NHIS) - Adult sample
We fielded continuous questions as categorical because of high item nonresponse.
Investigate other solutions in the IVR evaluation study
Note: The results of the NHIS evaluation study were presented at 2017 AAPOR and submitted for publication in the Journal of Survey Statistics and Methodology
NHIS Evaluation Study—Data collection metrics
Data collection for two separate one-week periods in 2017
Inbound calls RespondentsAAPOR4
response rate137,840 9,478 7.7%
Median interview length:
NHIS Evaluation Study— Comparison of demographic distributions among: population, RICS respondents and BRFSS respondents
RICS UWE : 1.25 /1.0 = 1.25
BRFSS UWE : 4.1 / 3.2 = 1.28
NHIS Evaluation Study—Night owls
31% of respondents are night owls
We define night owls as individuals who respond to the survey between 9pm and 9am.
NHIS Evaluation Study—Night owls (continued)
On average, compared to day-timers, night owls are:• More male• Younger• Less White-NH • Lower educatedThese are the groups that are underrepresented in outbound telephone surveys.
NHIS Evaluation Study— Calculating Sampling Weights
Demographic distributions for calibration: § Sex (2-levels)§ Age category (6-levels)§ Race/ethnicity (5-levels)§ Educational attainment (4-levels)§ Census division (9-levels)
We created delete-one jackknife weights (30 groups)
Base Weight
orPseudo-Inclusion
Probabilities
Calibrate to demographic distributions
NHIS Evaluation Study— Comparing categorical outcomes
NHIS Evaluation Study— Comparing continuous outcomes
NHIS Evaluation Study—Primacy Effect
If you get sick or have an accident, how worried are you that you will be able to pay your medical bills?
Order RICS Order 1 RICS Order 21 Very worried Not at all worried2 Somewhat worried Somewhat worried3 Not at all worried Very worried
Order RICS Order 1 RICS Order 21 Better About the same2 Worse Worse3 About the same Better
In regard to your health insurance or health care coverage, how does it compare to a year ago?
IVR Design Recommendations
Recommendations based on the IVR evaluation study:§ Allow continuous responses§ Use global “prefer not to answer” prompt§ Add a softball question before the first study question§ For the most important questions do not allow the
respondent to “barge-in”
Good Fits for RICS Methodology
RICS methodology with IVR data collection require:§ Short survey instrument (up to 35 questions)§ Simple questions
Target geographies comprised of an area code or groups of area codes
Minimal Screening Criteria§ Extensive screening increases costs
Examples of good fits for RICS§ Surveillance of population fleeing a natural disaster (Hurricane
Harvey)§ National or state surveillance of gastroenteritis (stomach flu)
Next Steps
Work in progress§ Develop the capacity to screen inbound callers with IVR and recruit
to a web-instrument§ Evaluate the stability of the estimates in a repeated cross-sectional
studies (to be presented at AAPOR 2018)
Future work§ Evaluate the best use of the IVR system
– Enter data with keypad – Conversational IVR– Or both
§ Evaluate bias in RICS surveys that use redirected toll numbers§ Develop and evaluate different weighting approaches
– Orthogonal to the demographic distributions used calibration– Correlated to the study outcomes.
