Make better surveysJeremy Rosenberg

@jeremy74

@jeremy74

About me

UX researcher & designer for 7 years

Agency and client side

BSc. in Psychology & Cognitive Neuroscience

MSc. in Human Computer Interaction (HCI)

Published psychology study in peer reviewed journal

Run many surveys!

@jeremy74

Surveys are hard

Face to face conversations are rich interactions

We seek feedback and modulate our responses

We can know when there is shared understanding

Surveys are rigid and prone to misunderstanding

@jeremy74

Is a survey the best methodology?

Avoid doing it instead of direct user contact e.g. interviews

Are there better ways to answer your question?

@jeremy74

Be clear about the purpose

Be clear about what you want to find out

What will you do with the information

Hypotheses

@jeremy74

A bad survey doesn’t smell

Often don’t know if the questions were bad/missed asking the right things

@jeremy74

Be ruthless with question choice

People will only answer so many questions

Justify every question

Respect the users’ time

Branch it

@jeremy74

@jeremy74

Don’t assume people can/want to answer

Adds noise to the data or increases drop-out

Allow skipping

Include active skipping e.g. n/a

Include ‘other’ option

@jeremy74

Put boring questions at the end

Improves completion rates

Completion tendency/sunk cost

Still get partial info for drop outs

Roberson, M.T. & Sundstrom, E. (1990). Questionnaire design, return rates, and response favorableness in an employee attitude questionnaire. Journal of Applied Psychology, 75, 354-357.

Martin, J.D. & McConnell, J.P. (1970). Mail questionnaire response induction: the effect of four variables on the response of a random sample to a difficult questionnaire. Social Science Quarterly, 51, 409-414.

@jeremy74

Be clear/specific

“How often do you cook?”

@jeremy74

Be clear/specific

“How often do you cook?”

“How often do you prepare a meal?”

@jeremy74

Be clear/specific

“How often do you cook?”

“How often do you prepare a meal?”

“How often do you prepare a meal, excluding ready-meals”

@jeremy74

Be clear/specific

“How often do you cook?”

“How often do you prepare a meal?”

“How often do you prepare a meal, excluding ready-meals”

“How often do you prepare an evening meal, excluding ready meals?”

@jeremy74

Be clear/specific

“How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?”

@jeremy74

Be clear/specific

“How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?”

“Did you prepare any evening meals, excluding ready-meals, in the last 7 days?”

then, if so,

“How many did you prepare?”

@jeremy74

Subjective questions are unreliable

Like usability testing, behaviour is more reliable than attitudes/preferences

Bertrand, M. & Mullainathan, S. (2001). Do people mean what they say? Implications for subjective survey data. American Economic Review, 91, 67-72.

@jeremy74

Anticipate biases

“Is sustainable seafood important to you”

versus

“Are you interested in any of the following?”

@jeremy74

Free text responses

Be judicious

Do you really have no idea of range of responses?

Free text as an answer option

@jeremy74

Other things to get right

Avoid leading questions

Avoid double barrelled questions

Avoid absolutes

Target the appropriate audience

Reduce reliance on memory

Consider cost/benefit of open ended questions

Map it out if it’s complicated

@jeremy74

Other things to get right

Consider a pilot study

Randomise question order, and answer options, where possible

Make sure it works on mobile

Map it out if it’s complicated

@jeremy74

@jeremy74

Other things to get right

Consider a pilot study

Randomise question order, and answer options, where possible

Make sure it works on mobile

Map it out if it’s complicated

@jeremy74

@jeremy74

Thanks Questions or tips to share?

Will Tweet link to slides @jeremy74