The Use of Random Digit Dialing in Household Surveys: Challenges and

Changes

Chris Chapman2008 IES Research Conference

Washington, DCJune 11, 2008

chris.chapman@ed.gov

Overview Two primary challenges in the use of traditional RDD collections for household surveys

• Declining response rates in traditional Random Digit Dialing (RDD) sample methodology

• Declining coverage rates in traditional RDD sample design studies

Analyses of alternative approaches for government RDD surveys

• Use of pre-phone contact mailed information materials

• Use of prepaid monetary incentives

• Study of staged, multimode collection methods

• Expanding coverage

National Household Education Surveys (NHES)

The analyses discussed today will be in context of work primarily done with NHES

NHES is a repeating household survey conducted by NCES

NHES has relied on RDD sampling approaches since its inception in 1991 through the latest collection in 2007

Sample is drawn from existing 100 banks starting with 130,000 to 475,000 numbers

Sample is restricted to non-business numbers and excludes cells phones

Topics AddressedEarly childhood education and nonparental care for infants, toddlers, preschoolers and students in early elementary grades

Parent and family involvement in the education of K-12 students

After school programs and activities for elementary and middle school children

Adult education for all civilian, non-institutionalized adults

Civic education and participation for K-12 students, parents and adults

School safety and discipline for K-12 students

National Household Education Surveys (NHES)

Typically relies on two stages of interviews

1. Screener interview to determine household and withinhousehold eligibility – generally need 45,000- 60,000 completed screener interviews per collection

2. Extended interviews collects detailed information about person(s) sampled within the household

Significant nonresponse problem in the Screener

1. Approximately 80% in the early 1990s

2. Approximately 50% now

Increasing Response Rates with RDD

Use of premailingConducted experiments showing that sending letters to households before calling helped improve first stage response rates

Use of monetary incentivesUse of small monetary incentives mailed to households before calling helped improve first stage response rates

In-person follow-upDuring recent bias study, found that in-person contact with phone interview refusal cases significantly boosted response rates

Premailing with RDDLetters about the purpose of the study and where to get information about it help boost first stage response rates

In the 2001 NHES, theresponse rate for matched households sent a letter was 75%response rate for matched households not sent a letter was 70%response rate for households that could not be matched was 55%

Challenge is matching randomly generated phone sample to mailing addresses – approximately 50 percent

See http://nces.ed.gov/pubs2005/2005071_2.pdf

Prepaid Incentives with RDD

In 2003, we tested monetary incentive options

Required that we had a mailing address – again had a 50% match rate

Tested no incentives against combinations of $2 and $5 incentives sent before the interview and as a refusal conversion technique

http://nces.ed.gov/pubs2006/2006066.pdf

Prepaid Incentives with RDD

42.1

38.9

42.4

42.1

40.9

45.2

46.6

45.4

47.1

69.9

69.7

69.1

68.9

67.9

66.7

63.7

69.5

67.3

64.148.8

30 35 40 45 50 55 60 65 70 75

10 - ($2/1st $2)

9 - ($5/1st $0)

8 - ($2/Priority $2)

7 - ($2/Priority $0)

6 - ($2/1st $0)

5 - ($0/Priority $2)

4 - ($0/Priority $0)

3 - ($0/1st $5)

2 - ($0/1st $2)

1 - ($0 brochure/1st $0)

Incentive group

Rate (%)

Unit responserate (%)

Ever refused rate (%)

adfafasfas adfafasfas

Prepaid Incentives with RDDOptimal approach for cost v. response rate effect was no initial incentive with a $5 incentive for refusal conversion

Next best approach was $2 initial incentive with a $2 conversion incentive

Both boosted first stage response rates over no incentives by approximately 5 percentage points

Tested use of priority and regular mail for initial and refusal conversion mailing - mailing did not have an independent effect beyond incentives for the first stage of data collection

Study of Potential Bias and Multimode Approach

Conducted an extensive bias test in 2007

Key components included:1. In person follow-up for telephone nonrespondents –

used to study potential nonresponse bias2. In person contact for households not in the RDD

frame – important to understand potential coverage bias

Multimode Work

Results not released yet but some results are generally clear

• If budget permits, in-person follow up can significantly increase response rates if used to help with telephone refusal conversion

• If not corrected, surveys based on RDD samples of land line phones alone can incure coverage bias in studies of prekindergarten through high school students and their families

Frame Adjustments

NHIS indicates that in 2007 16% of households were cell only• Approximately 14% of children live cell phone only households• Approximately 15% of adults live in cell phone only households

Can correct for some of coverage bias through weighting

Preferable to incorporate cell phones into frames though

• This may reduce overall response rates• Challenging for calculating proper weights

http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200805.pdf