questionnaire instructions and respondent behavior: a ... · 2007 mail and telephone survey 2011...

Questionnaire Instructions andRespondent Behavior:

A Cross-Survey Comparison

Brett McBrideDavid Cantor

Westat

AAPOR 67th Annual ConferenceMay 17 - 20, 2012 – Orlando, Florida

1

Research Question

What impact did changes in questionnaire instructions have on respondent behavior?

• Did revising skip instructions reduce skip errors?

• Did providing explanatory information closer to questions affect response distributions?

2

Outline

• Survey background

• Skip error calculation

• Findings on skip instruction format on error rates- Error rates by respondent characteristics

• Fruit and vegetable examples

• Findings on location of examples on distributions

• Summary and limitations

3

Survey Background

Health Information National Trends Survey (HINTS)

• Repeated cross-sectional survey of American adults’ access to health information

• Sponsored by National Cancer Institute

4

Survey Background

HINTS 32007 mail and telephone survey

3,582 respondents (mail)

31% response rate (mail)

Skip arrows for skip response

Example boxes at top of page

5

Survey Background

HINTS 3 HINTS 42007 mail and telephone survey 2011 mail survey

3,582 respondents (mail) 3,959 respondents

31% response rate (mail) 37% response rate

Skip arrows for skip response Arrow for skip and non-skip response

Example boxes at top of page Boxes placed next to questions

6

Calculation of Skip Error Rates

7

HINTS3 skip instruction

number missingnumber supposed to answer questionSkip omission rate =


8


number missingnumber supposed to answer questionSkip omission rate =

skip omission


9


number answeringnumber supposed to skip questionSkip commission rate =


10


number answeringnumber supposed to skip questionSkip commission rate =

skip commission


11



12

HINTS3 skip instruction HINTS4 skip instruction

Methods

• Compared filter questions (6)- Asked of all survey respondents - Similar question context across surveys

• Used replicate weights in WesVar to account for survey’s sampling method and changes in the sample composition over time

13

Findings: Reduced Skip Omission Rates

Q Filter Q description HINTS3 HINTS4 Prob. > |t|A2 Looked for health info (where) 1.3% 0.6% 0.06C6 #doctor visits last year (evaluate) 1.0% 0.7% 0.25D18 Smoked 100 cigs (current status) 2.2% 1.0% 0.04E2 Female (Pap test) 1.8% 2.6% 0.14H2 Diagnosed with cancer (type) 2.3% 0.9% 0.11K7 Foreign-born (year immigrated) 10.3% 6.4% 0.40

14

Findings: Reduced Skip Omission Rates

Q Filter Q description HINTS3 HINTS4 Prob. > |t|A2 Looked for health info (where) 1.3% 0.6% 0.06C6 #doctor visits last year (evaluate) 1.0% 0.7% 0.25D18 Smoked 100 cigs (current status) 2.2% 1.0% 0.04E2 Female (Pap test) 1.8% 2.6% 0.14H2 Diagnosed with cancer (type) 2.3% 0.9% 0.11K7 Foreign-born (year immigrated) 10.3% 6.4% 0.40

15

Findings: Change in Context

Different response category order, and differentfollow-up question for filter question on gender

16

HINTS3 HINTS4

Findings: Increased Skip Commission Rates

Q Filter Q description HINTS3 HINTS4 Prob. > |t|A2 Never looked for health info 6.6% 12.1% 0.08C6 No doctor visits last year 4.1% 21.2% 0.00D18 Never smoked 100 cigs 7.4% 12.5% 0.00E2 Male 6.1% 4.5% 0.23H2 Never diagnosed with cancer 0.2% 0% 0.01K7 US-born 1.1% 2.0% 0.06

17

Education Is Inversely Related to Omission Error

18

0%2%4%6%8%

10%12%14%

A2 C6 D18 E2 H2 K7

HINTS3

Less than HSHS GradCollege

Findings: Effect of Education Across Surveys

19

-6%

-4%

-2%

0%

2%

4%

6%

A2 C6 D18 E2 H2 K7

Change in Omission HINTS3 to HINTS4


Findings: Effect of Education Across Surveys

No indication of greater reduction for less-educated (or older respondents)

20

-6%

-4%

-2%

0%

2%

4%

6%

A2 C6 D18 E2 H2 K7

Change in Omission HINTS3 to HINTS4


Fruit and Vegetable Examples

• Moved boxes illustrating fruit and vegetable portions to be closer to questions

• Change in formatting of response options— HINTS3: double-banked response options— HINTS4: single-banked response options

21

Fruit and Vegetable Examples: HINTS3

22

Fruit and Vegetable Examples: HINTS4

23


24

0%

5%

10%

15%

20%

25%

30%

35%

None Less than0.5 cup

0.5 - 1 1 - 2 2 - 3 3 - 4 4+ cups

Daily Fruit Consumption

HINTS3


25

0%

5%

10%

15%

20%

25%

30%

35%


0.5 - 1 1 - 2 2 - 3 3 - 4 4+ cups

Daily Fruit Consumption

HINTS3HINTS4

* Chi-Square significance test, p=0.09


26

0%

5%

10%

15%

20%

25%

30%

35%


0.5 - 1 1 - 2 2 - 3 3 - 4 4+ cups

Daily Vegetable Consumption

HINTS3


27

0%

5%

10%

15%

20%

25%

30%

35%


0.5 - 1 1 - 2 2 - 3 3 - 4 4+ cups

Daily Vegetable Consumption

HINTS3HINTS4

* Chi-Square significance test, p=0.01

Findings: Fruit and Vegetables

• Post-hoc hypothesis of factors influencing change:1. Single-banked response options – increase perceived

midpoint versus HINTS3 double-banked options2. Respondents’ awareness and use of examples

decrease consumption patterns by signaling need for greater accuracy

• Effect of second factor out-weighed effect of first

28

Summary

• Skip arrows directing respondents to next question led to small reduction in skip omission errors

• Skip commission errors increased

• Overall, no differential change in skip omission rates by respondent education or age

• Some effect on response distributions by placing examples closer to response categories, reducing consumption patterns

29

Limitations

• Respondents’ answers determine type of skip errors that are possible

• Non-experimental design— Error rates across questions reflect different sample

groups (e.g., females, cancer survivors, foreign-born)— Change in example box location confounded by different

response option layout for fruit and vegetable questions— Could not control for actual trends over time

30

Thank You

Contact information:

Brett [email protected]

31

questionnaire instructions and respondent behavior: a ... · 2007 mail and telephone survey 2011...

Documents