Interviewer Affect and CARI EffectsLessons in Implementation and the Effects of CARI on aLarge-scale Longitudinal Study

AAPOR Annual Conference 2013

Ryan A. Hubbard

Overview

Medicare Current Beneficiary Survey (MCBS)implemented CARI with experienced interviewers andlongitudinal study respondents MCBS later introduced new interviewers and

respondents who had not been conditioned regardingCARI Discussion walks through the evolution of CARI on

MCBS and its impact on consent to record, interviewlength, data quantity and data quality

MCBS 15,000 Medicare beneficiaries in a rotating panel

design– 12 round study including baseline and exit interview– New panel introduced each fall– A panel retires each summer– Each round type has different supplements on healthCore CAPI interview on health care utilization

and expenditures– augments Medicare administrative data on events

and payments

MCBS QUESTIONNAIRE FLOW

4

CARI

A number of researchers have documentedCARI as a tool for:

– interview validation– assessment of interviewer performance– evaluation of data quality– question assessment

Notable sources include: Biemer, 2000, 2003; Herget, 2001; Thissen,2007, 2009; Fisher, 2012; Kinsey, 2012, Hicks, 2008, 2012.

MCBS CARI First used in Round 63, a summer round of MCBS on non-exiting cases

CARI data from rounds 63-65, a summer, fall, winter cycle

Requires consent of the respondent

Records selected items and captures corresponding screen with response

Interviewers cannot determine which items are recorded

Transmitted to home office and percentage of cases validated

Replaced traditional telephone validation for consenting CAPI respondents

Feedback on validation risks & poor interviewing technique to supervisors

CARI CONSENT

81%93% 94%

50%

60%

70%

80%

90%

100%

R63 R64 R65

CARI Consent by Round

N=10138 N=14331 N=13212

98%95%96%

92%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

New Interviewer Experienced Interviewer

R64 (second round) CARI Consent

New Sample

Existing Sample

DATA DIFFERENCES

31.0% 30.8%33.6%

46.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

2009 2010 2011 2012

RESPONDENT REFERED TO CALENDAR

CARI

Event Reporting

NEW EVENT REPORTING – ITEMS DESIGNED TO PROBE

0%

5%

10%

15%

20%

25%

30%

35%

2009 2010 2011 2012

HAD DENTALVISITS

SEE ANY HEALTHPRACTITIONERS

BUY OTHERMEDICALEQUIPMENT

BUY DIABETICEQUIPMENT

BUY EYEGLASSES,CONTACTS

BUY ORTHOPEDICITEMS

8.18 8.74

0

1

2

3

4

5

6

7

8

9

10

Prior to CARI CARI

Average Number of New Medical Events Reported (All Types)

7.27.47.67.8

88.2

8.4

8.6

8.8

9

6061

6263

6465

Average Number of New Medical Events Reported by Round

7.88.5 8.3

8.9 8.9 8.5

0

2

4

6

8

10

Summer Fall Winter

Average Number of New Medical Events by Round Type

Prior to CARI

CARI

50%

55%

60%

65%

2009 2010 2011 2012

ANY STATEMENTS FROM MEDICARE OR INSURANCE

Contact Information

QUANTITY OF NEW CONTACT INFORMATION WITH CARI

0%

5%

10%

15%

20%

25%

30%

35%

2009 2010 2011 2012

REPORTS A CONTACT 1(PREV. MISSING)

PHONE 2 FOR CONTACT2 UNDER ANOTHERNAME

REPORTS A CONTACT 2(PREV. MISSING)

NEED TO CHANGECONTACT 2 INFO

NEED TO CHANGECONTACT 1 INFO

SP HAS ANOTHER HOME

NEED TO CHANGEMAILING ADDRESS

2ND HOME ADDRESSCHANGED

Income and Assets

4.5%4.0%

3.3%

7.6%

2.3%

5.6%5.3%

0%

5%

10%

Income indollars

Value of Home How MuchOwed on Home

Total AssetValue

Total Amountof Asset Debt

Total Debt Medical Debt

Percentage Point Reduction in Missing Values for Income & Assets withCARI

DATA QUALITY

0.86 0.87

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Prior to CARI CARI

Proportion of New Medical Events Reported in Utilization

0.5

0.6

0.7

0.8

0.9

1.0

6061

6263

6465

Proportion of New Medical Events Reported in Utilization

CARI

INTERVIEW LENGTH

59.0

86.2

64.3

7.0

6.4

4.0

30

40

50

60

70

80

90

100

Summer (R63) Fall (R64) Winter (R65)

Additional Interview Length Attributable to CARI (minutes)

CARI

Prior to CARI

N=10138 N=14331 N=13212

11.9%

7.4% 6.2%

0%

5%

10%

15%

20%

25%

Summer (R63) Fall (R64) Winter (R65)

Percentage Increase in Interview Length Attributable to CARI

N=10138 N=14331 N=13212

Summary of Findings Patterns consistent with interviewer-induced effects

Initial consent issues appear to be related to interviewer resistance torecording

Interview length still up after accounting for questionnaire changes– A smaller but persistent CARI effect

Contact information and probes were shortcut areas for interviewers

Small impact on event entry in terms of:– Quantity of events– Area of instrument where events collected

While key data items see little change, operational item and itemsinterviewers may view as less important, potentially sensitive or hard to askshow a moderate but somewhat consistent CARI effect.

29

Limitations and Next Steps• The analysis of data quantity and quality would

benefit from matching event entry to administrativedata.

• Analyzing interviewer attributes as determinants ofconsent would assist in identifying problem areas andtraining.

• First look raises more questions than provides answers

• Findings should be woven into a more comprehensivemodel of interviewer effects based on that richliterature

Ryan A. HubbardRyanHubbard@westat.com

Thanks to additional Westat staff:Charles Halla

Wendy Hicks

Aaron Maitland

Andrew Mercer