insights on research design
DESCRIPTION
Colleen Roller's presentation from the UXPA Boston 2014 conference.TRANSCRIPT
Research Design
Gaining deeper insight
Colleen RollerAuthor & Practitioner
Areas of research
Behavioral decision theory The study of how people judge, decide and behave
Cognitive psychology The study of how people perceive, think, remember
and learn Social psychology
The study of how people are influenced and affected by other people
No neutral design
Preferences, judgments, and decisions are remarkably sensitive to the context in which they are made.
No neutral design
No neutral design
No neutral design
This presentation
Uses research findings to inform research design
Clever Hans
The power of the subconscious
Rosenthal/Jacobsen 1968
‘When we expect certain behaviors of others, we are likely to act in ways that make the expected behavior more likely to occur.’
- Rosenthal and Babad
Confirmation bias
The tendency to seek out and assign more weight to evidence that confirms one’s hypothesis, and ignore or underweight evidence that could disconfirm it.
So what?
We may be influencing participants – and research findings – in ways we’re not even aware of.
FRAMING
What is framing?
Framing
‘I’ve got great news. You managed to avoid a salary decrease.’
Frame the study
The purpose of this study is to test the design, not you.
Child custody
Parent AAverage incomeAverage healthAverage working hoursReasonable rapport with the childRelatively stable social life
Shafir, Simonson, Tversky, 1993
Which parent should get sole custody?
Award – 64%
Parent BAbove-average incomeMinor health problemsLots of work-related travelVery close relationship with the childExtremely active social life
Child custody
Parent AAverage incomeAverage healthAverage working hoursReasonable rapport with the childRelatively stable social life
Shafir, Simonson, Tversky, 1993
Which parent should not get sole custody?
Award – 64%Deny – 55%
Parent BAbove-average incomeMinor health problemsLots of work-related travelVery close relationship with the childExtremely active social life
Positive/negative
What did you like/dislike… How useful, or not,… How confident, or not… Why would you continue/stop using…
How we ask the question
Positive/negative Breadth/depth – broad/narrow
Did you notice anything different about this process…
How does this process compare to how you currently do this?
What information do you need in order to feel confident about doing x…
What would make you feel confident about doing x…
How often do you use the x website…
Tell me about how and when you use the x website…
Counterintuitive frame
What do you think of the brand?
What does the brand think of you?
‘A small question can never hold a large answer.’
- Gerald Zaltman (How Customers Think)
Assumptions
Be careful that your assumptions aren’t built into the question
Turn your assumptions into questions
Rating scales
How successful have you been in life, so far?
0 1 2 3 4 5 6 7 8 9 10Not satisfied Very satisfied
-5 -4 -3 -2 -1 0 1 2 3 4 5Not satisfied Very satisfied
34%
13%
Schwartz et al, 1991
Scholarships
Two scholarships are available. You prefer a scholarship with less waiting
time and more money. Which one would you pick?
Sun, Li, Bonini 2010
Graphical scale
Figure 1: wait time salient – preferred A Figure 2: $ amount salient – preferred B
Sun, Li, Bonini 2010
Graphical comparison
Toothpaste B
Toothpaste A
0 5 10 15 20 25 30 35
People (per 5000) with serious gum disease
E.R. Stone et al. 2003
Graphical comparison
Toothpaste A30 out of 5000
Toothpaste B15 out of 5000
E.R. Stone et al. 2003
So what?
How does the frame itself influence the meaning that people take away?
ORDERING OF QUESTIONS
Survey design
Please rate your overall satisfaction.
How many times have you had repairs?
German students
How happy are you these days?
How many dates did you have last month?
Strack et al, 1988
Reverse order
How happy are you these days? How many dates did you have last month?
How many dates did you have last month? How happy are you these days?
Strack et al, 1988
‘The sequential arrangement of information often creates the very meaning of that information.
The order of presentation determines the reaction.’
- Dr. Frank Luntz (Words That Work)
So what?
In what ways are you creating (unintended) meaning?
What are you measuring?
THE POWER OF WORDS
Different meanings
Gambling v. gaming Liberal v. progressive Liquor v. spirits Used v. pre-owned Etc.
An auto accident
At what speed did car A contact car B?
At what speed did car A smash into car B?
Loftus/Palmer 1974
31.8 mph
40.5 mph
Actual: 12 mph
Power of words
Make contact Smash into Crash into Bump into Run into Collide with
So what?
How are your words impacting user perception – and consequently, research
findings? how your business partners perceive the
findings?
Delivering findings
Users were… Aggravated… Annoyed… Confused… Disappointed… Frustrated… Irritated… Surprised…
Neutral
Users… Said… Expected… Thought… Anticipated…
IN SUMMARY
Research design
‘The function of a research design is to ensure that the evidence obtained enables us to answer the initial question as unambiguously as possible.’
The design matters
There is no neutral design
‘All research methods involve compromises with reality.’
- Gerald Zaltman (How Customers Think)
The solution
Be aware Control what we can
How we ask questions Ordering of questions Choice of words Assumptions and their effect Use of ‘scale’ – charts, graphs, rating scales Comparison – graphs, ratings
UXmatters.com UX magazine (uxmag.com)
Colleen Roller