experience research best practices - ux meet up boston 2013 - dan berlin

September 19, 2013

Dan BerlinExperience Research [email protected]@banderlin

Mad*Pow

EXPERIENCE RESEARCH BEST PRACTICES

Hi! I’m Dan Berlin @banderlin

BA in psychology from Brandeis U. Studied visual space perception

Seven years in technical support Sat as a participant for a usability study for a product I was working on Realized that user experience (UX) work is the perfect combination of computers and psychology

Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design

Two years at an interactive agency performing usability and neuromarketing research Then did some freelance UX consulting for about a year

Am now an Experience Research Director at Mad*Pow, an experience design agency based out of Portsmouth, NH

My passion is for research methodology and finding new ways to elicit data from participants

2

About this Presentation

Understanding your research goals Formative and evaluative studies

How to choose the right method(s) Methods chart

Gathering qualitative data Taking notes and organizing findings

3

UNDERSTANDING YOURRESEARCH GOALS

4

Understanding Your Research Goals

5

Preparing to prepare for your study

TechnologyUser

Needs

BusinessGoals

Your overall goal:

To uncover actionable business and design insights via user data

Start by understanding the business’ goals What is the overall goal of the interface? What are the interactions that drive the business? What are the important calls to action? Who are the target audiences? What do they want to know about their users? What do they think they already know about their users

How did they learn this?

Understanding Your Research Goals

6

Preparing to prepare for your study

Actionable = the data should indicate what exactly needs to change or be implemented in the interface to align it with users’ expectations and needs

This means you need to start with actionable (and attainable) goals, such as: How do perform a certain interaction today? Does the proposed IA align with user expectations? Are users able to complete a critical transaction? Do users understand the pricing schedule?

Goals will depend on the type of study Formative

Performed at the beginning of a project to learn how the project should proceed

Evaluative Performed during and towards the end of projects to determine if the design aligns with user expectations

Formative In-Person Usability Interviews Focus Groups Collaging Ethnography Surveys Diary Studies Card-Sorting

7

Evaluative In-Person Usability Remote Usability Unmoderated Usability Desirability Testing Eye-Tracking

Understanding Your Research GoalsStudy Types

HOW TO CHOOSE THERIGHT METHOD(S)

8

How to Choose the Right Method(s)

Make sure your research goals are clear Know what you want to learn Know what you will do with the information you gather Know what decisions the business needs to make

Know your constraints Timeline? Resources? Budget? Access to Users?

Create a Methods Chart The answers will become clear

9

Know Goals and Constraints


10

Method Pros Cons CandidateMethod 1 • List the advantages for

this method for this specific project

• List the disadvantages for this method for this specific project

Yes / No – Would this method be a good candidate for this specific project

Method 2Method 3Etc…

Methods Chart


Example Project Goals: Have 3 concepts for behavior change application. Which one is the best to

develop for maximum global appeal? Client wants answers as quickly as possible Client wants large numbers to provide confidence behind decision Client wants data collection in 5 countries around the world Designers want to know why or why not users chose each concept, to

provide additional design direction User population – adults who own a smart phone and want to get healthier

11

Methods Chart

Methods Chart Example

12

Method Pros Cons CandidateInterviews • Good for capturing

motivations for behavior change

• Can get good qualitative details on why users prefer each concept or not

• Good to explore issues to fine tune survey questions

• Can be done via phone (get broad geographic sample)

• Can be done quickly

• Small numbers – client wants large numbers

Y – would be good as a qualitative method to pair with a larger quantitative method

Focus Groups • Good for qualitative information gathering

• Could generate some interesting conversations about behavior change motivations

• Wouldn’t get as much detail as interviews

• Concern about group think when evaluating concepts

• Concern about not sharing details of personal goals in front of others

N – interviews would be better for qualitative

Survey • Good for large numbers• Easily replicated across

different countries/languages

• Can be done online for broad geographic distribution

• Can be done quickly

• Unclear what exact questions to ask

• Doesn’t provide detailed insights into qualitative topics

Y – good paired with qualitative method


Example Research Plan: Conduct telephone interviews in US with 12-15 participants Create online survey in US for 300 participants Use international research partners to conduct 12-15 interviews in 4

countries Use international research partners to conduct survey for 300 participants in

same 4 countries

Why this approach works: Interviews provide input into questions for survey Survey provides large numbers; interviews provide in-depth insight Both are commonly used methods and easy to replicate in different

countries Both methods can be done quickly

13

Methods Chart

GATHERING QUALITATIVE DATA

14

Gathering Qualitative Data

15

Usability Task Creation

Good usability study tasks are:

Non-leading: don’t give away the answer

Single tasks: don’t have users do two things in succession

Realistic: don’t have users do things they would not normally do

Self-explanatory: don’t have multiple sentences explaining the task

Achievable: always document what constitutes a “pass” for the task


16

Moderator’s Guide

A good study guide: Conveys the study goals and methodology to clients Serves as a quick reference to the moderator during the study Provides the moderator with a template to take hand-written notes, with ample space to

do so

The typical moderator’s guide contains:

Research method: a short paragraph explaining the usability study Study goals: a short bulleted list of the study goals Introduction: the moderator’s opening spiel when explaining the study to the participant Background questions: Typically demographic or product usage questions Tasks: The task, pass condition(s), notes for the moderator, and post-task questions Follow-up questions: The questions to be asked after all the tasks are complete


18

Notes Grid

Proper planning for taking notes is very underrated The organization and thoroughness of your notes will dictate the ease with which you

will create the final report Organized, complete notes = easy reporting Disorganized, incomplete notes = back to the video you go (ewww!)

Make your notes grid once your study & moderator’s guides are complete Give each question and subquestion its own row Put each participant in a new column (or vice-versa, if you like)

DO NOT put each participant/task in a new worksheet

Use data validation for quantitative data Task ease ratings, task success, multiple choice questions, etc. Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to

the question)

19

• Do not include the participant name in your notes grid

• Include a date/time cell to best align with the videos

• Visually separate sections of the study

• Fill data validated cells with a light color

• Hide columns when you have moved on to the next participant

Gathering Qualitative DataNotes Grid

20

Gathering Qualitative DataFindings Sheet

Scan the notes grid to determine findings Put these in a spreadsheet and assign and priority and category

In Conclusion

Set actionable and attainable research goals Know where you are in the project

Use a methods chart to weigh factors and choose a method Time, participants, goals, budget, etc.

Proper documentation = easier qualitative analysis Study guide notes grid findings spreadsheet report

21