random digit dialing cell phone surveys and surveillance systems: data quality, data weighting...
DESCRIPTION
TRANSCRIPT
Random digit dialing cell phone surveys and ill t d t lit d tsurveillance systems: data quality, data weighting strategies, and bias
Cristine D. Delnevo, PhD, MPH & Daniel A. Gundersen, MA,
UMDNJ‐ School of Public Health
Randal S ZuWallack MS & Frederica R Conrey PhDRandal S. ZuWallack, MS & Frederica R. Conrey, PhD
ICF Macro International
P t d t 137th A l M ti & E itiPresented at 137th Annual Meeting & ExpositionPhiladelphia, PA
November 7‐11, 2009
Work supported in part by the National Cancer Institute (R21CA129474 ) and a contract from the New Jersey Department of Health and Senior Services, through funding from the Cigarette Tax
Wireless substitution (US), 2005–2008Wireless substitution (US), 2005 2008
Source: Blumberg & Luke Wireless substitution: Early release of estimates from the National Health Interview Survey, July‐December 2008. National Center for Health Statistics. May 2009.
Demographic differencesDemographic differences• Gender: men more likely than
b i l lWireless substitution by age (and time)
women to be wireless only
• SES: Adults living in (30.9%) and near poverty (23.8%) more likely h hi h i d l
Wireless substitution by age (and time)
than higher income adults (16.0%) to be wireless only
• Region: Wireless substitution hi h t i S th (21 3%) dhighest in South (21.3%) and Midwest (20.8%) vs. Northeast (11.4%) or West (17.2 %)
R /Eth i it Wi l• Race/Ethnicity: Wireless substitution highest among black (21.4%) and Hispanic (25 0%) adults vs Non Hispanic(25.0%) adults vs. Non‐Hispanic white adults (16.6%)
Source: Blumberg & Luke Wireless substitution: Early release of estimates from the National Health Interview Survey, July‐December 2008. National Center for Health Statistics. May 2009.
Biased health estimates?Biased health estimates?
• Potential for biased health estimates due to samplePotential for biased health estimates due to sample under‐coverage remains a real, growing threat to RDD health surveys
• Cell‐phone only also differs with respect to health behaviors and the validity of some health estimates based on traditional RDD surveys are increasingly questionable
Health estimates by phone statusHealth estimates by phone status
Has a landline Wireless-only No telephonetelephone
y p
NHIS July – December 2007
5+ alcoholic drinks in 1 day 17.7 37.3 27.1Current smoker 18.0 30.6 38.6Uninsured 13 7 28 7 44 1Uninsured 13.7 28.7 44.1Has a usual place for care 87.5 68.0 61.8
Flu vaccination 32.7 16.6 20.9
Ever tested for HIV 34.7 47.6 45.8
Source: Blumberg SJ, Luke JV. Coverage bias in traditional telephone surveys of low‐income and young adults. Public Opin Q. 2007;71:734–749
Cigarette smoking among young adults, 2003 2005 NHIS & BRFSS2003‐2005, NHIS & BRFSS
Source: Delnevo, Gundersen & Hagman (2008) Declining prevalence of alcohol and smoking estimates among young adults nationally: artifacts of sample under‐coverage? Am J of Epidemiology
New challenges: Wireless Mostly?
Telephone Status, NHIS July‐December 2008
• The percentage of adults living in wireless‐mostly households has been increasing
• Who are they?• Who are they? (demographic & health behaviors?)
• Will they respond to landline surveys?
Source: Blumberg & Luke Wireless substitution: Early release of estimates from the National Health Interview Survey, July‐December 2008. National Center for Health Statistics. May 2009.
Reactions?Reactions?
• Starting in 2009, the Center f Di C l dfor Disease Control and Prevention (CDC) is requiring states to incorporate cell phoneincorporate cell phone interviews in their regular BRFSS sample
• Yet there is no widely yaccepted methods of evaluating data quality or data weighting, particularly for state and local areafor state and local area surveys.
• AAPOR Cell Phone Task Force reportForce report
This special session:This special session:
• Analysis of data quality of cell phone• Analysis of data quality of cell phone surveys
• Demonstrate weighting procedures for merging cell phone samples with landline samples
• An assessment of bias in landline only• An assessment of bias in landline only surveys, and
Assessing Data Quality ofAssessing Data Quality of Cell Phone Random Digit g
Dial Surveys
Frederica R ConreyRandy Zuwallack
Data Quality
Q lit f R• Quality of Responses• Quality of Sample
2008 New Jersey Adult Tobacco Survey
R d Di it Di l L d (RR 19%) d• Random Digit Dial Land (RR=19%) and Cell Phone (RR=16%)
• Short version– 49 Questions (min=37; max=70)– 534 Cell completes– 468 Landline completes
Survival Analysis
Diff t l t diff t• Different people get different surveys because of skip patterns
• Survival analysis– Measures the impact of survey mode on non-
response– Controls for differences in survey length
Quality of Responses
Item Non-Response
M M di Std
Failure Rates by Phone ModeMean Median Std
Cell 3.3% 2.2% 7.4%
Landline 3.4% 2.0% 5.5%
Survival predicted by phone mode:p y pHazard=1.00, p>.95
Open Ended Response
O d t f thTotal Responses per Open
Open end reports of the events of 2 recalled commercials
Responses / OE DK / OE
p p pEnd by Phone Mode
commercials– 251 respondents were
asked at least one
/ OE DK / OE
Cell .52 .49
Landline .48 .55
open ended question P .56 .33
Quality of Sample
Unit Non-Response
D t lit i th t d ifData quality is threatened if– Response rates are low AND– The people who DO NOT respond are
different from those who DO.If cell phone respondents are less likely to
respond, then there is non-response bias.
Survival Analysis
C ll d l dli d t t• Cell and landline respondents may get different surveys
• Response rates alone don’t tell the story• Survival analysis tells whether cell y
respondents are more likely to break off given the same survey lengthg y g
Survival Model
25%
20%
25%
g O
ff
10%
15%
e B
reak
ing
Cell
5%
10%
Sam
ple
Landline
0%0 20 40 60 80
Survey Questions 001Survey Questions p<.001
What does a difference in survey survival mean?
C ll d t it th l dli• Cell respondents quit sooner than landline respondents.
• The sample under-represents cell phones• The longer the survey, the worse the g y
nonresponse bias• The solution?The solution?
– Careful weighting– Short surveys– Short surveys
In a population study of tobacco use
Mi i l diff S b t ti l diff
behavior…
Minimal difference between cell and landline in response
Substantial difference between cell and landline in responselandline in response
quality• No difference emerged in
landline in response rate
• Sample quality may beNo difference emerged in item-nonresponse
• No difference emerged in i h f d
Sample quality may be threatened if cell phone surveys are too long or weighted incorrectlyrichness of open end
responses.weighted incorrectly
Weighting Cell Phone SurveysWeighting Cell Phone Surveys
Randal Zuwallack
Frederica R ConreyFrederica R Conrey
Thanks
• Cris DelnevoCris Delnevo
• Dan Gundersen
C ( 2 C 29 ) f• NCI (R21CA129474), New Jersey Dept of Health and Senior Services
Dual Frame
A Adults in landline households with no cell phone
D
A. Adults in landline households with no cell phone,
B. Adults in landline households with a cell phone, and
C. Adults in non‐landline households with a cell phone (cell only).
Dual Frame
Common designs:
D
Common designs:
Dual frame w/ no overlap: Landline (A+B) + Cell (C)
Dual frame w/ overlap: Landline (A+B) + Cell (B+C)
Uncommon design:
Dual frame w/ no overlap: Landline (A) + Cell (B+C)Dual frame w/ no overlap: Landline (A) + Cell (B+C)
Weighting Challenges
• Challenge 1: How do we put the dual framesChallenge 1: How do we put the dual frames together?
• Challenge 2: Differential Nonresponse
Challenge 1
• How do we put the dual frames together?How do we put the dual frames together?• No overlap
– Estimate of cell‐only population size?Estimate of cell only population size?• Internal estimate• External estimate: NHIS (Blumberg et al.)
• With overlap– Must determine group membership– Adjust for multiple selection probabilities– Estimate of phone group population sizes?
Cell Survey
• “In addition to your cell phone is there atIn addition to your cell phone, is there at least one telephone inside your home that is currently working and is not a cell phone? Docurrently working and is not a cell phone? Do not include telephones only used for business or telephones only used for computers or faxor telephones only used for computers or fax machines.” – ‘yes’ = dual user while those who responded– yes = dual user, while those who responded
– ‘no’ = cell‐only
Landline Survey
• “In addition to your residential landlineIn addition to your residential landline telephone, do you also use one or more cell phone numbers?”phone numbers? – ‘yes’ = dual user
‘no’ = landline only– no = landline only.
Example 1‐‐Colorado
• Combine with BRFSSCombine with BRFSS
• Group membership K f ll– Known for cell
– Unknown for landline
• Limited to dual frame w/ no overlap– Used 15% (NHIS state estimates) for merging landline and cell
– Poststratified dual sample to age and sex.
Example 2
• You are midway through a landline survey andYou are midway through a landline survey and want to add cell phones. You don’t know who has a cell phone and who doesn’t What arehas a cell phone and who doesn t. What are your options?1) Add cell only1) Add cell only
2) Add cell and dual‐users
Challenge 2
• Differential NonresponseDifferential Nonresponse
• Cell‐only overrepresented when conducting cell phone surveyscell phone surveys. – Contact rate
C i– Cooperation rate
• Those who rely more on their cell phone will be easier to reach.
Telephone Reliance
L Cea
ndli
ell
P
ne
hone
Landline Frame
L Cea
ndli
ell
PLandline householdsne
hone
Landline sample
Cell Phone Frame
L Cea
ndli
ell
PCell Phone Usersne
hone
Cell sampleCell sample
Dual Frames
Landline sample Landline sample
Cell sample Cell sample
Dual Frame sample Dual Frame sample
Ideal Realistic
Our Goal
R b lRebalance on cell reliancereliance
Rebalance on landline reliance
Measuring Telephone Reliance
• Cell only landline onlyCell only, landline only
• Classify Dual users“Of ll th t l h ll th t i ”– “Of all the telephone calls that you receive, are…”
• All or almost all calls received on a cell phone? (cell‐mostly)mostly)
• Some received on a cell phone and some on a regular landline phone? (true‐dual)
• Very few or none received on a cell phone? (landline‐mostly)
Telephone Reliance
L Cea
ndli
ell
P
LandlineMostly(1)
TrueDual(2)
CellMostly
LandlineOnly(0)
CellOnly
ne
hone
(1) (2) (3)(0) (4)
Response Propensity
• Adjust for differential nonresponse byAdjust for differential nonresponse by benchmarking against NHIS
• Logistic regression model• Logistic regression model– Dependent: Survey type
1 b ll i ti l ll h• 1 = observe cell user in national cell phone survey
• 0 = observe cell user in NHIS
Independent: Cell phone reliance (1 4) age race– Independent: Cell phone reliance (1‐4), age, race
Data sources
National cell sample NHISNational cell samplen=500
NHIS
Landline mostly 8% 23%
True dual 27% 42%
Cell mostly 23% 17%
Cell only 42% 18%
Cell users 100% 100%
Applying the model
• Applied to same data—poststatificationApplied to same data poststatification
• Applied to independent dataA ti ti l ll l th– Assumption: national cell sample measures the odds ratio for observing a cell‐only respondent in a cell sample relative to a dual‐usera cell sample relative to a dual user.
S l l• State, local surveys
Applying the model
ColoradoColoradoCell Sample
w/o NR adj w/ NR adj
Landline mostly (1) 9% 26%
True dual (2) 26% 37%
C ll tl (3) 19% 15%Cell mostly (3) 19% 15%
Cell only (4) 46% 21%
Total cell sample 100% 100%
Applying the model
• Assume landline only is 20% (we don’t know)Assume landline only is 20% (we don t know)
CO
Landline only 20%
Landline mostly 21%
True dual 30%
Cell mostly 12%NHIS
Cell only 17%
Total population 100%
NHIS state
estimate = 15%p p
City Sample
LandlineSample
Cell Sample Combined Samples
w/o NR w/ NR w/o NR w/ NRw/o NR adj
w/ NR adj
w/o NR adj
w/ NR adj
Cell‐only ‐ 43.5 18.9 35.5 13.4
Cell‐mostly 12.4 25.1 19.8 13.8 12.2
True Dual 30.1 23.6 35.5 18.7 25.1
Landline‐mostly 19.1 7.9 25.8 9.9 17.2Landline mostly 19.1 7.9 25.8 9.9 17.2
Landline‐only 38.4 ‐ ‐ 23.9 32.1
Conclusions
• Dual‐frameDual frame– There are ways to combine the data, even when we don’t have a full picture of group membershipwe don t have a full picture of group membership.
• Differential nonresponseResponse propensity model rebalances the cell– Response propensity model rebalances the cell sample based on cell reliance.
Can be applied at state and local levels when no– Can be applied at state and local levels when no benchmarks exist.
– Next steps: explore a response propensity modelNext steps: explore a response propensity model for landline.
Examining the bias in landline only g ysurveys: How does the cell phone only population differ from the landlinepopulation differ from the landline
population on health indicators, and are estimates from landline surveys biased?
Daniel A. Gundersen, MA, UMDNJ‐SPH
Cristine D. Delnevo, PhD, MPH, UMDNJ‐SPH
Randy S. ZuWallack, MS, ICF Macro
Cell Phone Substitution and RDD surveysCell Phone Substitution and RDD surveys
• RDD surveys (e.g. BRFSS) have traditionally only sampled household telephones (i.e. landlines)
• Up until early 2000s, rate of cell phone only households was small
• From mid 2000s, rate of substitution has grown substantially– 6 7% of adults in 2005 to 18 4% in 2008 nationally16.7% of adults in 2005 to 18.4% in 2008 nationally
• Higher among certain demographic groups1– Young adults– Hispanics and Blacks– Poor and near poor
What is bias due to coverage error in the li f ?sampling frame?
• Non‐covered population is different fromNon covered population is different from covered population on some variable of interestinterest
– If proportion of non‐covered ( ) is small, bias will be small
– If difference between the covered and
noncovered is small, bias will be small,
Previous ResearchPrevious Research• Data from Jan 2004‐June 2005 NHIS found2
– Greater than 1 percentage point bias in binge drinking, smoking prevalence, usual place for medical care, receiving influenza vaccine
• Data from 2007 NHIS found3Data from 2007 NHIS found– Bias increased slightly for past year binge drinking and receiving influenza vaccineThese biases were larger among young adults and low income– These biases were larger among young adults and low income persons
• Data from 2001‐2005 BRFSS on 18‐24 year olds found4– Prevalence of binge drinking, heavy drinking, and cigarette smoking declined during 2003‐2005; coincided with large increase in wireless substitution among young adults
– NHIS and NSDUH did not observe similar declines during this period
Our Study:Our Study:
• Objective:Objective: – Assess the presence of bias in landline RDD due to exclusion of cell phone only on select healthexclusion of cell phone only on select health indicators
• Data Source and Instrument:Data Source and Instrument:– Cell phone RDD of adults in Colorado (n=501)
• May to September 2008May to September 2008
• Instrument was shortened version of BRFSS
– BRFSS from same data collection period (n=4,527)BRFSS from same data collection period (n 4,527)
MethodologyMethodology
• Cell Phone sample:Ce o e sa p e:– Design weights account for probability of selection
• BRFSS– Standard BRFSS design weight accounts for strata, number of landlines and adults in the household
f d b ( )* *– Postratified by age(7)*sex*race
• Merged dataD i i ht l d t t h f– Design weights scaled to represent share of population by phone status
– Postratified by age(7)*sex*racey g ( )
Statistical AnalysesStatistical Analyses• Comparisons of BRFSS landline and Cell Only based on design
weights• Comparison of BRFSS landline and merged data are
postratified to demographic makeup of CO
• We assume the merged data to be unbiased (i.e. no coverage error due to cell only exclusion)error due to cell only exclusion)
• All analyses conducted in STATA v.10.1 to account for complex sampling design
Table. CO BRFSS vs. CO Cell Only, May‐September 2008 (n=5,028)y, y p ( , )
BRFSS landline Cell only Differencey
Health Indicator % (95%CI) % (95%CI) % (95%CI)
* ( ) ( ) ( )Smoking* 15.29 (±1.14) 28.14 (±4.46) ‐12.85 (±4.60)
Ever had HIV test* 36.64 (±1.85) 52.51 (±5.09) ‐15.87 (±5.41)
Having health insurance* 88.36 (±1.11) 72.46 (±4.38) 15.9 (±4.51)
Having primary care provider* 85.91 (±1.16) 60.39 (±4.85) 25.52 (±5.00)
Not affording care due to cost* 12 27 (±1 08) 20 42 (±3 94) ‐8 15 (±4 08)Not affording care due to cost 12.27 (±1.08) 20.42 (±3.94) ‐8.15 (±4.08)
*p<.05; data weighted to correct for sampling design
Figure 1. Bias of Smoking Prevalence, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008
7 5
10
5
7.5
0
2.5
‐2.5
0
‐7.5
‐5
‐10
Figure 2. Relative Bias of Smoking Prevalence, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Only RDD (n=501) May‐September 2008
50%
30%
40%
0%
10%
20%
‐20%
‐10%
0%
50%
‐40%
‐30%
‐50%
Figure 3. Bias of Ever had an HIV test, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008
10
5
7.5
2.5
‐2.5
0
‐7.5
‐5
‐10
Figure 4. Relative Bias of Ever had an HIV test, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Cell Only RDD (n=501) May‐September 2008
50%
30%
40%
0%
10%
20%
‐20%
‐10%
0%
50%
‐40%
‐30%
‐50%
Figure 5. Bias ‐ Having health insurance, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008
10
5
7.5
2.5
5
‐2.5
0
‐7 5
‐5
‐10
‐7.5
Figure 6. Relative Bias Having Health Insurance, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Cell Only RDD (n=501) May‐September 2008
50%
30%
40%
0%
10%
20%
‐20%
‐10%
0%
50%
‐40%
‐30%
‐50%
Figure 7. Bias Has primary care provider, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008
10
5
7.5
2.5
5
‐2.5
0
7 5
‐5
‐10
‐7.5
Figure 8. Relative Bias Has Primary Care Provider, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Cell Only RDD (n=501) May‐September 2008
50%
30%
40%
0%
10%
20%
‐20%
‐10%
0%
50%
‐40%
‐30%
‐50%
Figure 9. Bias ‐ could not afford health care due to cost, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008and CO Cell Only RDD (n=501) May‐September 2008
10
5
7.5
2.5
‐2.5
0
‐7.5
‐5
‐10
7.5
Figure 10. Relative Bias ‐ could not afford health care due to cost, CO BRFSS (n 4 527) and CO Cell Only RDD (n 501) May September 2008BRFSS (n=4,527) and CO Cell Only RDD (n=501) May‐September 2008
50%
30%
40%
50%
10%
20%
‐20%
‐10%
0%
‐40%
‐30%
‐50%
Summary of FindingsSummary of Findings
• Bias is present not only among those with high wireless substitution rates
• Smoking prevalence underestimated among those with higher wireless substitution rateshigher wireless substitution rates
• Ever had an HIV test substantially underestimated among all groups
R l ti bi l th ith hi h i l– Relative bias large among those with high wireless substitution rates (young adults, non‐whites, low SES)
• Bias for health care insurance and having primary care id i d i d hi bprovider is underestimated among non‐whites, but
overestimated among other groups– Relative bias is small
Implications for study design and lanalysis
• When possible, include an RDD of cell phone only population (BRFSS now does this)– If you can’t, be aware of the potential for bias and interpret
findings accordingly• If you’re analyzing landline RDD data from past years
– Interpret findings with potential bias in mind– Historical trend may observe artificial changes due to coverage
error– Wireless substitution rates differ by geographic region so
problem may be less in certain areasA bi d b h hi i ll i• A bias present today may not be the same historically or in the future– Characteristics of the early adopters may not be the same as the
current cell only population today or laggardscurrent cell only population today or laggards
LimitationsLimitations
• Unable to assess bias in some subpopulationsUnable to assess bias in some subpopulations due to small sample size
• Study does not account for cell phone mostly• Study does not account for cell phone mostly population
ReferencesReferences
1. Blumberg SJ & Julian V. Luke. (2009). Wireless Substitution: Early l f ti t f th N ti l H lth I t i Srelease of estimates from the National Health Interview Survey,
July‐December 2008.2. Blumberg SJ, Luke JV & Marcie L. Cynamon. (2006). Telephone
coverage and health survey estimates: evaluating concern aboutcoverage and health survey estimates: evaluating concern about wireless substitution. American Journal of Public Health. 96(5): 926‐931.
3. Blumberg SJ & Julian V. Luke. (2009). Reevaluating the need for concern regarding noncoverage bias in landline surveys. American Journal of Public Health. 99(10): 1806‐1810.
4. Delnevo CD, Gundersen DA & Brett T. Hagman. (2008). Declining estimated prevalence of alcohol drinking and smoking amongestimated prevalence of alcohol drinking and smoking among young adults nationally: artifacts of sample undercoverage? American Journal of Epidemiology. 167(1): 15‐19.
Contact InfoContact Info
Cristine Delnevo PhD MPH delnevo@umdnj eduCristine Delnevo, PhD, MPH [email protected] A. Gundersen, MA [email protected]
Randal ZuWallack [email protected] Conrey [email protected]