evp, global client servicesinsightinnovation.org/wp-content/uploads/2015/06/pdf/... · 2015. 7....

SURVEYSI

SURVEYABANDONED

It’s Not Me, it’s you!

Presentation by: Melanie CourtrightEVP, Global Client Services

Research Participants and Data Speak on Mobile Design and Data Quality

©2015 Research Now Group, Inc.

Our Relationship with Survey Participants

Marketing research is not just a one-way monologue. It’s a relationship that is based on mutual trust and respect.

Researchers can’t take participants for granted.

2


Our Relationship with Survey Participants

Marketing research is not just a one-way monologue. It’s a relationship that is based on mutual trust and respect.

Researchers can’t take participants for granted.

It’s time we start listening.

3


Research Promises … The Reality

Promises We Make to Participants

What We ActuallyDeliver to Participants

4


Are We Earning Participant Trust and Respect?

Promises We Make to Participants

Better user experience for survey takers overall

Mobile optimization of surveys

Device agnosticity

What We ActuallyDeliver to Participants

5


Why They Care About Our Mobile Promises

10%

27%

15%

45%

0

10

20

30

40

50

Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q12012 2013 2014 2015

Mobile in Research

+300%(vs. Q2 2012)

+270%(vs. Q2 2012)

Participants are using their smartphone for research more than

they used to

6


Field Experiment Study Overview

PC-Style Single Select Mobile Optimized

PC Smartphone Tablet Smartphone Tablet Smartphone TabletDevices Shown to

Survey topics: Carbonated beverage purchasing behaviors & advertising and marketing attitudes, and general lifestyle attitudes and behaviors

Fielded to a census-rep panel balanced on inbound clicks: Participants recruited from the panels of Research Now.

Manipulated variables: Scale type & length

Measured variables: Device type (PC, smartphone, tablet), Speeding, Straight-lining/ variability, Stylistic responding, Extreme Response Style (ERS), Midpoint Response Style (MRS)

Survey Grid Types

7


Participant Reactions: Mobile Grids

• Tough to use

• Too big for the screen

• Need to scroll

• Small for touch accuracy

• Difficult to maneuver

• Unable to see grid headers

Participant Grid Problems“The biggest challenge with the survey are the

grid questions.”

“I didn’t like the grids that didn’t fit on the screen. I would suggest having the top rows and

the side information not scroll so as you scroll around you still see that information”

“It was kind of hard to see the full scale

on my phone.”

“I did have to scroll up to see the column heading to make sure I was

giving the correct response.”

“It was hard to scroll and be sure I had the right row corresponding to the question I

was answering.”

8


Participant Reactions: Mobile Grids

• Data Accuracy Issues

• Reduced Representativeness

• Difficulty Retaining Participants

What does it mean?“The navigation on my screen was too complicated.”

“I had to go back to see the question again, and then go up to the very top to see the answer on

the grid to know which was accurate.”

“I find it hard to get my finger inside that

one little small circle.”

“The survey felt long and repetitive and scrolling back and forth was

not ideal.”

“It just wasn’t 100% clear which button I had just selected.”

9


7.8

10.1

14.8

31.8

22.8

12.7

9.1

11.3

13.5

8.4

45.0

11.2

8.9

16.8

27.6

16.3

8.6

10.1

10.6

19.6

9.8

PC Style

Single select

Mobile Optimized

PC Style

Single select

Mobile Optimized

PC Style

Tabl

etSm

artp

hone

PC

PC Style on mobile is the slowest by far …

Scal

e Ty

pe 10-point

7-point

5-point

(Minutes)

PC

Smartphone

Tablet

except when it makespeople speed!

“I sometimes get tempted to rush through grid questions”

“Sometimes I don’t [answer accurately] because they get

repetitive and there are too many.”

10


94.2%

93.8%

94.4%

94.0%

94.6%

95.0%

93.5%

92.9%

93.6%

94.5%

94.7%

91.9%

95.2%

92.4%

45.8%

8.6%

46.8%

45.7%

8.9%

46.9%

44.1%

PC Style

Single select

Mobile Optimized

PC Style

Single select

Mobile Optimized

PC Style

Tabl

etSm

artp

hone

PCTen-Point Scales Are Hard to Complete

Scal

e Ty

pe 10-point

7-point

5-point

“It was a bit confusing jumping

to the next question when you answer on a smartphone.”

PC

Smartphone

Tablet

Note: Scales are converted to percentages for ease of comparison across scale length

Especially in Single-Select Style!

The easy way to mobile

“optimization” isn’t the best

11


80.5%

82.2%

82.9%

82.4%

83.4%

84.1%

82.3%

81.4%

81.2%

83.5%

84.5%

83.6%

83.0%

82.3%

80.2%

68.3%

83.1%

82.1%

68.8%

83.3%

78.3%

PC Style

Single select

Mobile Optimized

PC Style

Single select

Mobile Optimized

PC Style

Tabl

etSm

artp

hone

PCLeast Amount of Love for 10-point Single Select

Scal

e Ty

pe 10-point

7-point

5-point

PC

Smartphone

Tablet

Note: Scales are converted to percentages for ease of comparison across scale length

Especially 10 point Single-Select Style!

“I'm not really fond of the grid type questions.”

“Very difficult to complete on

smartphone ... prefer tablet or anything

with a larger screen.”

12


Easy & Enjoyable Surveys = Better Data Quality

0.31

-0.002

-0.273

0.585

-0.081

-0.361

Variability Extreme Response Midpoint Response

Pear

son

Corr

elat

ion

coorelation with "survey was easy"coorelation with "survey was enjoyable"Correlation with “Survey was easy”Correlation with “Survey was enjoyable”

When people enjoyed the survey, and found it easier, their answers show more variability, and

they’re less likely to use midpoint.

“I think directionally I gave the right answers, so when it was 1 through 10 if I was going for a 3 or 4 I was close, but not exactly

right on all of them.”

13


Data Quality Drops After the First Bad Grid

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Grid #1 Grid #2 Grid #3 Grid #1 Grid #2 Grid #3

Extreme Response Midpoint Response

Purchase Likelihood

Attitudes

Psychographics

10-point

7-point

5-pointScal

e Ty

pe

Extreme Response Midpoint Response

14


46.9%

43.8%

47.1%

48.3%

44.5%

46.2%

45.0%

44.8%

44.0%

41.0%

46.2%

43.0%

41.5%

40.4%

42.0%

38.0%

40.7%

41.2%

42.6%

39.8%

41.5%

PC Style

Single select

Mobile Optimized

PC Style

Single select

Mobile Optimized

PC Style

Tabl

etSm

artp

hone

PCExtreme Responding: Overall Avg. By Condition

Scal

e Ty

pe 10-point

7-point

5-point

PC

Smartphone

Tablet

15


19.6%

20.1%

19.5%

19.2%

20.8%

21.6%

21.0%

10.3%

9.2%

10.3%

10.6%

11.9%

12.5%

11.9%

8.7%

9.6%

9.3%

7.8%

9.2%

9.8%

9.2%

PC Style

Single select

Mobile Optimized

PC Style

Single select

Mobile Optimized

PC Style

Tabl

etSm

artp

hone

PCMidpoint Responding: Overall Avg. By Condition

Scal

e Ty

pe 10-point

7-point

5-point

PC

Smartphone

Tablet

16


Consumer Insights

Corporate Strategy

ProductLifecycle

Imagine a World Where …• Single data source ensures consistent methodology and quality• Relationships with your customers & your competitors’ customers• Opportunity to consolidate data spend (i.e. savings)

Data lives together …

17


Imagine a World Where…

“I believe that the data will set you free. At the end

of the day it’s about how you turn those pieces of

information into insights that will improve business.”

--- Steven Rice

…and is insightful.

18


Key Takeaways• #MRX is a two-way dialogue.

• Stop taking participants for granted.

• The survey taker isn’t an anonymous statistic.

• Even if you don’t care about the participant (shame!), you should care about the data.

• Every choice we make affects the data.

• Be creative. Be smart. Be considerate.

“I took my first survey ever using my smartphone, and it

was not as pleasant an experience as when I take

surveys online. I don’t think I’ll be doing those surveys on my smartphone anymore.”

19

The concepts presented in this presentation are the property of Research Now.Duplication or dissemination of the information without the express written consent of Research Now is prohibited.

Thank you!

Melanie CourtrightEVP, Global Client Services

[email protected]

June 2015

evp, global client servicesinsightinnovation.org/wp-content/uploads/2015/06/pdf/... · 2015. 7....

Documents