understanding survey design and the psychology of survey ... · five principles for design of paper...
TRANSCRIPT
![Page 1: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/1.jpg)
Melissa Chapman Haynes, Ph.D.
MESI Spring Training 2016
Understanding Survey Design
and the Psychology of Survey
Response to Write Better
Surveys
![Page 2: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/2.jpg)
“It says they
want our
feedback.”
![Page 3: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/3.jpg)
Introductions
![Page 4: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/4.jpg)
![Page 5: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/5.jpg)
Bad data is not better than no
data
![Page 6: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/6.jpg)
Learning Objectives
Describe principles of survey design
Understand the psychology of survey response
Identify & revise problematic survey items
Identify strategies for increasing response rates
![Page 7: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/7.jpg)
Purpose
Population
Timing
Mode
Use
Principles of Survey Design
![Page 8: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/8.jpg)
Purpose
Population
Timing
Mode
Use
Principles of Survey Design
![Page 9: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/9.jpg)
When should I use a survey?
![Page 10: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/10.jpg)
• Gather self-reported information on knowledge, behaviors, beliefs, experiences
• Population is defined
• Limited or well-defined need for detailed information
• Anonymity possible
Purpose: When is a survey the
appropriate method?
![Page 11: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/11.jpg)
Item Types
![Page 12: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/12.jpg)
Likert-type (rating scale)
Dichotomous (T/F or Y/N)
Multiple choice, ranking, rating scales
Closed-ended
![Page 13: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/13.jpg)
![Page 14: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/14.jpg)
![Page 15: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/15.jpg)
![Page 16: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/16.jpg)
![Page 17: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/17.jpg)
Use: When uncertain of entire range of
alternative answers or when you want
to gather examples, stories, lists, or
descriptions
Open-Ended Items
![Page 18: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/18.jpg)
![Page 19: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/19.jpg)
![Page 20: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/20.jpg)
Purpose
Population
Timing
Mode
Use
Principles of Survey Design
![Page 21: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/21.jpg)
![Page 22: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/22.jpg)
The importance
of sampling
![Page 23: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/23.jpg)
Who is your sample?
Be informed about and sensitive to needs
and constraints of your sample How to best reach a homeless population?
How to best reach young adults?
Are there cultural considerations (e.g.,
gender, ethnicity, geography, etc.)?
![Page 24: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/24.jpg)
Purpose
Population
Timing
Mode
Use
Principles of Survey Design
![Page 25: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/25.jpg)
Timing
![Page 26: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/26.jpg)
Purpose
Population
Timing
Mode
Use
Principles of Survey Design
![Page 27: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/27.jpg)
Paper
Phone
Online
![Page 28: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/28.jpg)
Impersonality
Cognitive Burden
Legitimacy
Data
Quality
![Page 29: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/29.jpg)
Seven
Design
Principles
to Reduce
Cognitive
Burden
Take advantage of knowledge that is
available externally + long-term memory
Simplify the structure of tasks
Make both controls and actions that perform
visible to the user
Rely on natural mappings between actions
and their consequences
Exploit physical and cultural constraints
Allow for errors
Rely on standardization when other design
principles do not apply
![Page 30: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/30.jpg)
![Page 31: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/31.jpg)
Paper Surveys
Self-Administration
Impersonality
Accuracy
Level of Reporting
Rate of Missing Data
Cognitive Burden
Reliability
![Page 32: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/32.jpg)
Online Surveys
Online
Legitimacy
Accuracy
Level of Reporting
Cognitive Burden
Rate of Missing Data
Reliability
![Page 33: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/33.jpg)
Adaptive
Design
![Page 34: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/34.jpg)
Five
Principles
for Design
of Paper
or Online
Surveys
1. Consistently use graphical elements (e.g.,
contrast and spacing) to define a clear path
through the questionnaire.
2. Use prominent visual guides to redirect
respondents when conventions within a
questionnaire must change.
3. Place directions where they are easily seen
and close to where they are needed.
4. Keep separate pieces of information
physically close when they must be connected
to be understood.
5. Ask only one question at a time.
![Page 35: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/35.jpg)
Consistency
Section I Strongly
Agree
Agree Disagree Strongly
Disagree
No
Response
I am satisfied with X.
I found the content to be useful.
Section II Strongly
Disagree
Disagree Agree Strongly
Agree
No
Response
I would attend X event again.
I would recommend X to a friend.
![Page 36: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/36.jpg)
Aesthetic
![Page 37: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/37.jpg)
Telephone Surveys
Telephone
Legitimacy
Accuracy
Level of Reporting
Cognitive Burden
Rate of Missing Data
Reliability
![Page 38: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/38.jpg)
Purpose
Population
Timing
Mode
Use
Principles of Survey Design
![Page 39: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/39.jpg)
Use of Survey
Results
![Page 40: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/40.jpg)
In the past 7 days, on how many
days did you eat breakfast?
How tall are you in feet and inches?
What is your current weight (lbs)?
![Page 41: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/41.jpg)
![Page 42: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/42.jpg)
![Page 43: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/43.jpg)
What
demographic
categories
might make
sense for this
survey study?
![Page 44: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/44.jpg)
So how can I integrate
purpose, planning, timing,
mode and use into my survey
planning?
![Page 45: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/45.jpg)
Pulling it all
Together:
Planning
Survey Design
![Page 46: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/46.jpg)
1. Review survey &
scenario
2. Work backward
to create a survey
plan
3. Critique the
survey
![Page 47: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/47.jpg)
BREIF SCENARIO:
You are hired as the external evaluator for
Company X in June 2015. They want you to conduct
an evaluation of Subordinate Appraisal of
Management (SAM) within various departments of
Company X. You have six months to conduct the
entire evaluation (which will consist of a survey).
There are approximately 5,000 people in your
population.
![Page 48: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/48.jpg)
• Describe your experience in trying to create
a planning document.
DISCUSSION QUESTIONS
• Were there any surprises during the
process? Outstanding questions?
• What surprised you the most about trying to
improve this survey?
• If this survey had been implemented as is,
what issues might there have been with the
data collected?
![Page 49: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/49.jpg)
Psychological Response Process
Response Processes for Question Type
Factual questions
Dates and durations
Frequencies
Attitude questions
Psychology of Survey Response
![Page 50: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/50.jpg)
Statistical Error
Cognitive Error
![Page 51: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/51.jpg)
Psychological Response Process
Comprehension
Retrieval
Judgment
Selection
Response Processes for Question Type
Psychology of Survey Response
![Page 52: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/52.jpg)
Avoid
confusion
and chaos
Comprehension
Retrieval Judgment
Selection
![Page 53: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/53.jpg)
Comprehension
Retrieval Judgment
Selection
![Page 54: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/54.jpg)
Response
Breakdown:
Comprehension
Attend to questions and
instructions
Represent logical form of
questions
Identify question focus
(information sought)
Link key terms to relevant
concepts
![Page 55: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/55.jpg)
• Syntactic Difficulties
• Semantic Effects
– Presupposition
– Unfamiliarity
– Vagueness
• Survey Pragmatics
Comprehension
![Page 56: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/56.jpg)
- Asking leading questions
- Unbalanced scales
- Asking double barreled
questions
- Using words and phrases
that may have differential
meanings to respondents,
jargon, abbreviations, and
slang
• All possible response options
(closed-ended items)
• Option of No Response (NR),
Don’t Know (DK), or No Opinion
(NO)
• Response categories that are
mutually exclusive
Avoid… Be sure to include…
![Page 57: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/57.jpg)
Please tell me whether you
would vote for or against a
candidate who supports
reducing federal spending on
education and welfare?
![Page 58: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/58.jpg)
Does it seem possible or does
it seem impossible to you that
the Nazi extermination of the
Jews never happened?
![Page 59: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/59.jpg)
Do you own a tablet PC?
![Page 60: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/60.jpg)
How much do
you love this
presentation?
Love it
Like it
It’s pretty good
![Page 61: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/61.jpg)
Rules of
Thumb
• Keep questions simple
• Avoid complicated syntax
• Decompose questions that cover multiple
possibilities into simpler questions that
cover a single possibility apiece
• Define ambiguous or unfamiliar terms
• Avoid vague concepts, and provide
examples when such concepts must be
mentioned
• Replace vague qualifiers with ranges that
specify exact probabilities, frequencies,
etc. (as necessary)
![Page 62: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/62.jpg)
Comprehension
Retrieval Judgment
Selection
![Page 63: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/63.jpg)
7 ± 2
![Page 64: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/64.jpg)
Response
Breakdown:
Retrieval
Processes
Generate retrieval
strategy and cues
Retrieve specific,
generic memories
Fill in missing
details
![Page 65: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/65.jpg)
Memory
for Events:
Reference
Periods
Example 1: What is your age? Example 2: Do you sometimes drink more than you should? Example 3: Have you worked at a job or business at any time during the past 12 months?
![Page 66: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/66.jpg)
Other
Recall
Factors
Proximity More distant memory More inference
Distinctiveness Multiple, similar events More difficult
to distinguish
Emotional Impact Strong emotional impact stronger
memory
![Page 67: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/67.jpg)
Variable affecting recall Problem Implications for Design
Time of Occurrence Longer ago = harder
recall
Shorten the reference period
Proximity to temporal
boundaries
Events near
boundaries are
easier to recall
Use personal landmarks, life events
calendars
Distinctiveness More distinct =
easier recall
Use multiple cues to single out
events;
Tailor length of reference period to
properties of the target events Emotional impact Important emotional
events = easier
recall
Summary of Factors Affecting Recall
![Page 68: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/68.jpg)
When you were growing up,
how frequently did your father
attend religious services?
![Page 69: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/69.jpg)
Comprehension
Retrieval Judgment
Selection
![Page 70: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/70.jpg)
Response
Breakdown:
Judgment
Assess completeness and
relevance of memories
Draw inferences based on
accessibility
Integrate material retrieved
Make estimate based on
partial retrieval
![Page 71: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/71.jpg)
How many days during the
past 12 months has this
condition kept him in bed all
or most of the day?
![Page 72: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/72.jpg)
A working mother can
establish just as warm and
secure a relationship with her
child as a mother who does
not work. Strongly agree/Agree/Disagree/Strongly disagree/ Don’t know
![Page 73: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/73.jpg)
What type of insurance plan
do you have? Blue Cross/Blue Shield / Medica/ Health
Partners / etc.
![Page 74: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/74.jpg)
Visual Information Heuristic
Consistent with
Heuristic
Mildly Inconsistent Strongly Inconsistent
Strongly agree Strongly agree It depends
Agree Agree Strongly agree
It depends Disagree Strongly disagree
Disagree Strongly disagree Agree
Strongly disagree It depends Disagree
![Page 75: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/75.jpg)
Comprehension
Retrieval Judgment
Selection
![Page 76: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/76.jpg)
Response
Breakdown:
The response
itself
Map judgment
onto response
category
Edit response
![Page 77: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/77.jpg)
Psychological Response Process
Response Processes for Question Type
Factual questions
Dates and durations
Frequencies
Attitude questions
Psychology of Survey Response
![Page 78: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/78.jpg)
Factual Questions
Retrieval Process Judgments may involve:
• Adopting strategy to
retrieve info from long-term
memory
• Using retrieval cues to
trigger recall
• Filling in partial memories
through inference
• Completeness or accuracy
of retrieval
• Inferences that fill in gaps
of what is recalled
• Integration of pieces of
information into a single
overall response
![Page 79: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/79.jpg)
Dates and Durations
![Page 80: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/80.jpg)
During the last 6 months, did
anyone steal things that
belonged to you from inside
ANY car or truck, such as
packages or clothing?
![Page 81: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/81.jpg)
Durations
Judgments may involve:
• Forward telescoping – reporting of vents that took
place before the specified reference period
• Incorrect inferences
– Memory (in)accessibility
– Incorrect guesses
– Uncertainty about date ranges
– Rounding
Frequencies
![Page 82: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/82.jpg)
During the past 12 months,
about how many times did ___
see or talk to a medical
doctor?
![Page 83: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/83.jpg)
Attitude Items
![Page 84: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/84.jpg)
a. Math is one of my worst subjects.
b. I am confident that I can learn math.
c. Math is difficult for me.
…
Reverse score negative items
![Page 85: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/85.jpg)
Summarize
Comprehension
Retrieval Judgment
Selection
![Page 86: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/86.jpg)
Planning Survey Design and Content
Identifying Problematic Survey Items
Revising Problematic Survey Items
Problematic survey items
![Page 87: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/87.jpg)
![Page 88: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/88.jpg)
Planning Survey Design and Content
Identifying Problematic Survey Items
Revising Problematic Survey Items
Problematic survey items
![Page 89: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/89.jpg)
Planning
Survey Design
and Content
![Page 90: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/90.jpg)
![Page 91: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/91.jpg)
Cognitive
Interviews
![Page 92: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/92.jpg)
Simple example:
- Turn to a neighbor
- Decide who will be the
interviewer/interviewee
![Page 93: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/93.jpg)
How many days in the past 7
days have you eaten fruit?
![Page 94: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/94.jpg)
How many days in the past 7
days has anyone smoked in
your work area?
![Page 95: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/95.jpg)
Planning Survey Design and Content
Identifying Problematic Survey Items
Revising Problematic Survey Items
Problematic survey items
![Page 96: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/96.jpg)
Critiquing a Survey’s Content and
Format
Individual and Small Group Activity
![Page 97: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/97.jpg)
![Page 98: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/98.jpg)
On average, how many minutes do your
students participate in physical activities
per day?
![Page 99: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/99.jpg)
1. What do you believe the question is asking?
2. What do the specific words mean to you? Which words are open to interpretation/have multiple meanings?
3. What information do you need to be able to recall to answer the question?
4. Do the response categories match your own internally generated answer (as applicable)?
Discussion Questions
![Page 100: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/100.jpg)
• What surprised you the most about trying to improve this survey?
• If this survey had been implemented as is, what issues might there have been wit the data collected?
• What did you learn about survey design from this activity?
Large Group Debrief
![Page 101: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/101.jpg)
What is an “Optimal” Response Rate
Strategies for Increasing Response Rates
Strategies for increasing
response rates
![Page 102: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/102.jpg)
• Estimating non-response bias
• Use – how will the results be used? What are
the stakes?
What is a “good enough” response
rate?
![Page 103: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/103.jpg)
Pre-Incentives
![Page 104: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/104.jpg)
![Page 105: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/105.jpg)
Graduate Student Survey
![Page 106: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/106.jpg)
![Page 107: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/107.jpg)
Make a list of possible
incentives that could be used
to obtain a high survey
response rate
![Page 108: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/108.jpg)
Incentive Respondent
Population
Positive Aspects
or Advantages
Negative
Aspects of
Disadvantages
![Page 109: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/109.jpg)
• How did you decide what would be an effective
survey incentive? What were your criteria?
• What issues arose in your discussions concerning
the positive and negative aspects of various
incentives?
• What ethical issues did your group discuss relative
to the use of incentives?
• Under what circumstances might the use of
incentives be inappropriate?
![Page 110: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/110.jpg)
The Psychology of Survey Response by Tourangeau, Rips, & Rasinski
Cognitive Interviewing: A Tool for Improving Questionnaire Design by Willis
Cognitive interviewing: A how-to guide, http://www.uiowa.edu/~c07b209/interview.pdf
Question taxonomy,
http://mnemosyne.csl.psyc.memphis.edu/QUAID/question_taxonomy.htm
Dillman, D., Smyth, J.D., and Christian, L.M. (2009). Internet, mail, and mixed-mode
surveys: The tailored design method, 3rd edition. Hoboken, NJ: Wiley
Biemer, P.P. (2010). Total survey error: Design, Implementation, and Evaluation.
Public Opinion Quarterly, 74 (5), pp. 817-848.
![Page 111: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/111.jpg)
AAPOR T-shirt Slogan Contest Past Winners: “I lost my validity at AAPOR” “AAPOR: Freqs and Geeks” “Public Opinion Research: Fighting the war against error” “Trust us – we’re 95% confident” “If you don’t like the estimate, just weight”
![Page 112: Understanding Survey Design and the Psychology of Survey ... · Five Principles for Design of Paper or Online Surveys 1. Consistently use graphical elements (e.g., contrast and spacing)](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f16b8883d32d03b6044e41a/html5/thumbnails/112.jpg)
The PDA team