ordering your attention: response order effects in parallel phone … · 2015-01-28 · ordering...
TRANSCRIPT
![Page 1: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/1.jpg)
Ordering Your Attention:
Response Order Effects in Parallel
Phone and Online Surveys
Frances M. Barlas ICF International
&
Randall K. Thomas
GfK Government and Academic Research
Presented at the 67th Annual Conference of AAPOR
Orlando, FL
May 17 – May 20, 2012
![Page 2: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/2.jpg)
2
Introduction
• The order in which response options are presented
plays a role in respondents’ answer selection
process and has long been shown to affect survey
results (Schuman and Presser, 1981)
• Response order effects occur when respondents
select responses based on the order in which the
responses are presented:
o Primacy effects – higher selection of options when
they are presented early in the response set
o Recency effects – higher selection of options when
they are presented later in the response set
![Page 3: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/3.jpg)
3
Introduction
• To help understand when and why response
order effects occur researchers have
considered a number of factors including:
oRespondents’ memory and attention span
oRespondents’ age and education
oResponse format
oMode of response presentation
![Page 4: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/4.jpg)
4
Introduction
• Krosnick and Presser (2010) examined a
number of past studies on order effects
among categorical response options and
observed a pattern:
o Primacy effects are more likely to occur when
alternatives are presented visually
o Recency effects are more likely to occur when
response alternatives were presented orally
![Page 5: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/5.jpg)
5
Introduction
• Response-order elaboration model -
– Order of response presentation affects the amount
that respondents are able to pay attention to and
process each response.
– Explains mode differences in response order
effects:
• Items presented visually: the degree to which
respondents consider earlier items likely interferes
with their ability to fully consider later items.
• Items presented orally: consideration of earlier
response options is disrupted by the presentation of
the next response option
![Page 6: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/6.jpg)
6
Introduction
• Response-order effects have often been used to
explain response selection with collections of
categorical responses, but less is known about
selection of ordered responses as occurs with scales.
• Scales have an inherent order and connection
between responses –
– Presentation of one response can lead to a rapid
understanding of the nature of the scale.
– Often the nature of the scale is evident from the
question stem.
![Page 7: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/7.jpg)
7
Introduction
• Order of responses may have less of a profound
effect when dealing with scales compared to
categorical response options due to the automated
way in which scales are processed.
• Our interest in this study was to compare a set of
questions asked in parallel phone and web surveys
that used the same response formats to examine the
strength and direction of order effects.
![Page 8: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/8.jpg)
8
Method
Two surveys were fielded in parallel in April 2006:
1. Online survey –
• 3,937 completed interviews
• drawn from the Harris Poll Online panel based on a stratified, random selection on age, sex, region, education, income, and race strata to resemble U.S. general population.
2. Telephone survey –
• 1,008 completed interviews
• Random digit dial sampling within sampling blocks having at least 2 working numbers
• Targeting landlines, not cell phones
![Page 9: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/9.jpg)
9
Method
Questionnaire:
• There were 27 items fielded in parallel across the two modes with a response order manipulation
• Three types of items were used:
–Political evaluation items – bipolar rating scale
–Economic confidence items – bipolar rating scale
–Expenditure likelihood items – unipolar rating scale
• Response order manipulation:
–Respondents randomly assigned to order of response presentation
– ‘Most to Least’ or ‘Least to Most’
–Assignment was independent within each of the questionnaire sections
![Page 10: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/10.jpg)
10
Method
Political Evaluation Items
• Three approval rating items
- Approval of President George W. Bush’s handling of job as president
- Approval of President George W. Bush’s economic policies
- Approval of President George W. Bush’s defense policies and military decisions
•Responses: Strongly approve, Somewhat approve, Neither approve nor disapprove, Somewhat disapprove, Strongly disapprove
•Online response options presented vertically, one item per screen.
![Page 11: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/11.jpg)
11
Method
Political Evaluation Items – Random assignment to 5 of 8 possible performance evaluation items
1. Democrats in Congress
2. Republicans in Congress
3. Senate Majority Leader Bill Frist
4. House Speaker Dennis Hastert
5 . Vice President Dick Cheney
6 . Secretary of State Condoleezza Rice
7. Secretary of Defense Donald Rumsfeld
8. Attorney General Alberto Gonzales
• Responses: Excellent, Pretty good, Only fair, or Poor
• Online items were presented in grid format, all items presented on a single screen.
![Page 12: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/12.jpg)
12
Method
Economic Confidence Items
• Random assignment to 4 of 7 possible items.
• Online items presented all on same screen with response options displayed horizontally
1. How would you rate the economic condition of your region of the nation? [Very good, Somewhat good, Neither good nor bad, Somewhat bad, Very bad]
2. How would you rate the current job market of your region of the nation? [Very good, Somewhat good, Neither good nor bad, Somewhat bad, Very bad]
3. How would you rate the economic policies of the national government? [Very good, Somewhat good, Neither good nor bad, Somewhat bad, Very bad]
![Page 13: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/13.jpg)
13
Method
Economic Confidence Items (4 of 7 assigned)
4. Compared to a year ago, how has your household’s financial condition changed? [Improved a lot, Improved somewhat, Remained the same, Worsened somewhat, Worsened a lot]
5. What are your feelings concerning the economy of your region of the nation over the next 6 months? [Very optimistic, Somewhat optimistic, Neither optimistic nor pessimistic, Somewhat pessimistic, Very pessimistic]
6. How do you think that the job market in your region of the nation will change over the next 6 months? [Much better, Somewhat better, Remain the same, Somewhat worse, Much worse]
7. Thinking about your household’s current financial situation, do you expect it to be better or worse in the next 6 months? [Much better, Somewhat better, Remain the same, Remain the same, Somewhat worse, Much worse]
![Page 14: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/14.jpg)
14
Method
Expenditure Likelihood Items
• Random assignment to 5 of 9 possible items
• Online items presented as a grid, with items presented in rows and responses in columns, on a single screen.
![Page 15: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/15.jpg)
15
Method
Expenditure Likelihood Items – likelihood in the next 6 months (5 of 9 assigned):
1. Buy or lease a newly manufactured car, truck, or van
2. Move to a different residence
3. Purchase a house or condo
4. Take a vacation costing more than $1000
5. Have more money to spend the way you want
6. Save or invest more money
7. Start a new business
8. Buy a boat or recreational vehicle (e.g., trailer, motor home)
9. Buy a new computer
• Responses: Not at all likely, Somewhat likely, Likely, Very likely, Absolutely certain
![Page 16: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/16.jpg)
16
Results
• For analyses, all items were range-adjusted to be equivalent on a 0 to 1 scale to allow for greater comparability between scales of different types.
• We first conducted separate analyses for each item.
Phone Online
Number of experiments 27 27
Primacy Effects 18 16
Statistically Significant 3 4
Recency Effects 9 11
Statistically Significant 2 7
![Page 17: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/17.jpg)
17
Results
• Next we compared means by mode and response order, after controlling for age, sex, income, education, region of the country, and race/ethnicity.
• Looking across all 27 items, there was a significant mode effect (Phone M=.357; Online M=.273; p<.001), which replicates other findings of reduced positive ratings given online.
• There was not a significant response order effect, but a marginally significant Mode X Order interaction with some tendency for a primacy effect for phone surveys for these scales.
![Page 18: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/18.jpg)
18
Results – Overall Means
Analysis of overall means controls for age, sex, income,
education, region of the country, and race/ethnicity covariates.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
Least to Most Most to Least
Phone
Online
![Page 19: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/19.jpg)
19
Results –Means for Political Items
(bipolar scale)
Analysis of overall means controls for age, sex, income,
education, region of the country, and race/ethnicity covariates.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
Least to Most Most to Least
Phone
Online
![Page 20: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/20.jpg)
20
Results –Means for Economic Items
(bipolar scale)
Analysis of overall means controls for age, sex, income,
education, region of the country, and race/ethnicity covariates.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
Least to Most Most to Least
Phone
Online
![Page 21: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/21.jpg)
21
Results –Means for Expenditure
Likelihood Items (unipolar scale)
Analysis of overall means controls for age, sex, income,
education, region of the country, and race/ethnicity covariates.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
Least to Most Most to Least
Phone
Online
![Page 22: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/22.jpg)
22
Results – Pooled Analysis
The pooled analysis, combining all items within each specific item section showed few response order effects:
• The only significant response order effect (p<0.05) for the telephone administration occurred in the political items section, a primacy effect.
• With the online administration, there were significant differences with the political items (recency effect) and expenditure likelihood items (primacy effect), but not the economic items.
![Page 23: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/23.jpg)
23
Results - Validity
• We used the 18 items that reflected political and economic evaluations as predictors of political party ID, using a regression approach to allow us to statistically evaluate the impact of mode and response order on validity.
• Political party identification was used as the primary dependent variable for validity purposes since it used a different measurement methodology than the measures of interest, a branching strategy – first asking for party identification, with a follow-up asking strength of identification and for independents the direction they lean. The branching strategy randomly alternated the presentation of ‘Democrat’ and ‘Republican’.
![Page 24: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/24.jpg)
24
Results - Validity
• We found that the effect of Mode was significant (p<.001), with online measurement having significantly higher correspondence between the political and economic measures with political party identification.
• Response Order was also significant (p<.001) with the ‘Least to Most’ order having higher validity in both modes of survey administration.
![Page 25: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/25.jpg)
25
Results: Predictive Validity- Adjusted R2
Predicting Party ID with Political and Economic Items
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Telephone Online
Least to Most
Most to Least
![Page 26: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/26.jpg)
26
Conclusions
• The effects of response order appeared to be modest at best in both online and telephone modalities for the items we studied.
• There was no clear pattern of primacy and recency effects by mode of survey administration, as has been observed with categorical response options.
• Means on the phone were found to be higher generally, replicating other findings that respondents are more likely to use positive categories in evaluations with a human interviewer.
![Page 27: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/27.jpg)
27
Conclusion
• There was some evidence to support that presenting categories in a ‘Least to Most’ fashion may be associated with higher levels of validity for both telephone and online surveys.
• These findings may be qualified by the nature of the scales we used – most were evaluative bipolar scales and most were presented in a grid format online.
• More research is needed to compare response order effects with different types of scales and to further investigate the impact of mode and response order on measurement validity.
![Page 28: Ordering Your Attention: Response Order Effects in Parallel Phone … · 2015-01-28 · Ordering Your Attention: Response Order Effects in Parallel Phone and Online Surveys Frances](https://reader030.vdocuments.site/reader030/viewer/2022040610/5ed053bc3a066a18461b5aba/html5/thumbnails/28.jpg)
28
Thank You!
For further information: