Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

Sharon Shavitt

Department of Business Administration

Survey Research Laboratory

BA 531: Survey Methods

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

A Survey is…

A set of cognitive tasks

A social encounter (a conversation)

Survey Orientation

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

3. Formatting the response

4. Editing the response

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

3. Formatting the response

4. Editing the response

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

– Understand the literal meaning

– Infer the intended meaning

• Conversational logic

• Response alternatives

• Question context

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

Where do you live?

Understanding/Interpreting the Question

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

Where do you live?

Literal meaning is clear, but inference about intended meaning will be strongly influenced by interview location.

Understanding/Interpreting the Question

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. How satisfied are you with this course?

2. How satisfied are you with this curriculum in general?

Context Effects: Interpreting the Question

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. How satisfied are you with this course?

2. How satisfied are you with this curriculum in general?

Don’t ask specific questions before general ones (e.g., rate course, then curriculum). Respondent assumes answer to 2nd question should be nonredundant with 1st.

Context Effects: Interpreting the Question

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

3. Formatting the response

4. Editing the response

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

Some questions request an evaluation of a target of judgment. What is your expected grade in this course?

Rate the instructor’s overall teaching effectiveness.

Evaluations are strongly influenced by context (making

criteria salient, setting a standard of comparison, etc.)

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

What is your expected grade in this course?

Rate the instructor’s overall teaching effectiveness.

Evaluations are strongly influenced by context (making criteria salient, setting a standard of comparison, etc.)

Evaluation

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

Some questions require retrieving instances from memory or making a

computation about the frequency of some event.

( How often do you…)

Retrieving or Computation

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

Longer or Shorter is Better?

How many times do you study in a typical week?

How many times do you study in a typical week?

This might include studying at the library,

reading a textbook while on a bus or train,

looking at the course website, or going over your

notes after class.

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

3. Formatting the response– Nature of a frequency scale can dramatically affect

responses. How to address this?

– Should we use broad labels? (“often,” “sometimes,” “seldom” ): Annie Hall

– Vague quantifiers: Provide evaluative as well as frequency information. Cannot disentangle.

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

• How many hours do you watch TV each week?

– Less than 1 hour

– Between 1-3 hours

– Between 3-5 hours

– Between 5-7 hours

– More than 7 hours

Autobiographical Reporting: Formatting the Response

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

• How many hours do you watch TV each week?

– Less than 7 hours

– Between 7-9 hours

– Between 9-11 hours

– Between 11-13 hours

– More than 13 hours

Autobiographical Reporting: Formatting the Response

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

1. Understanding and interpreting the question

2. Generating an opinion or judgment

3. Formatting the response

4. Editing the response– Should we ask threatening questions multiple

times to enhance reliability?– How can we make it OK to say yes (or no)?– Are shorter questions better than longer ones?

Respondent’s Tasks

Context Effects: Interpreting the Question

• Should we ask specific questions before general ones? (e.g., rate course, then curriculum)

• What about with attitude questions?

• Does nature of the frequency scale affect responses?

Did you vote in the recent election?

Voting is not necessarily convenient for every person. Did you happen to vote in the recent election?

Threatening Questions: Response Editing