ux poland user research workshop 04.21.15

04 • 21 • 2015

UX Poland 2015 | #charmingresearchUser Research Workshop

This material contains confidential and proprietary information of Charming Robot LLC and may not be reproduced, distributed or disclosed, in whole or in part, without its express consent.

©2015 Charming Robot LLC. All rights reserved.

WHO ARE WE?

2

USABILITY RESEARCH WORKSHOP

Matt Lee Dan Maccarone@DanMaccarone@MLeeConsulting

Hear us both talk about UX/User Research on this podcast: storyinabottle.charmingrobot.com

http://storyinabottle.charmingrobot.com



WHAT WE WILL DO DAY 1

3

9:30am - 10:15am

10:15am - 11:15am

11:15 - 11:30pm

11:30pm - 11:45pm

12:00pm - 4:30pm

5:00pm - 5:30pm

Part 1: The Why, Who, and What of User Research

Part 2: The Interview: Plan, Process, & Practice

Break

Part 3: Team Creation: Teams and planning

Part 4: Let’s RESEARCH

Part 5: Feedback: Day 1 learnings and questions


THE WHO, WHY, & WHAT

THE INTERVIEW

THE FINDINGS

Q&A




4


9:30am - 10:30am

10:30am - 2:30pm

2:30pm - 4:30pm

4:30pm - 5:00pm

5:00pm - 5:30pm

—-

Part 6: Findings: Drawing Conclusions & Taking Action

Part 7: Team Analysis

Part 8: Presentations

Part 9: Feedback and Outcomes

Part 10: Q & A


THE INTERVIEW

THE FINDINGS

Q&A

PART 1

The Who, Why, & What of User Research



WHAT WE’LL COVER‣ Why research matters

‣ Choosing the right research for your goals

‣ When should you do research?

‣ Identifying who you should talk to

‣ Developing a screener

‣ Recruiting

6



THE INTERVIEW

THE FINDINGS

Q&A



WHY RESEARCH MATTERS

7


YOU ARE NOT THE USER99.9% of the time


THE INTERVIEW

THE FINDINGS

Q&A



WHY USER RESEARCH MATTERS‣ Develop understanding & empathy for customers

‣ Dispel assumptions that team may have and replace with facts and data

‣ Reveal new opportunities

‣ Unearth unknown problems

‣ Help support hypothesis about customers & their needs

8



THE INTERVIEW

THE FINDINGS

Q&A



THE RESEARCH PROCESS

‣ Opportunity

‣ Validate

‣ Internal conflict

9


IDENTIFY GOALS WHO TO TALK TO CHOOSE METHOD DO RESEARCH SYNTHESIZE FINDINGS

1 - 2 WEEKS 2 - 3 WEEKS 2 - 3 WEEKS 1 - 2 WEEKS

‣ Screeners

‣ Recruiting

‣ Incentives

‣ Discussion guide

‣ Create test cases

‣ Interviews

‣ Logistics

‣ Best practices

‣ Analysis

‣ Insights

‣ Reports

‣ Presentations

‣ Concepts


THE INTERVIEW

THE FINDINGS

Q&A



WHEN IS RESEARCH DONE?‣ Considering new market opportunities

‣ Evaluate features already in market

‣ When internal teams are not aligned on features & priorities

‣ Research should be a continuous part of your process

10



THE INTERVIEW

THE FINDINGS

Q&A



MAPPING RESEARCH TO PRODUCT DEVELOPMENT

11


PRODUCT STRATEGY

EXPERIENCE & VISUAL DESIGN

TECHNOLOGY DEVELOPMENT

‣ Concept testing

‣ Prototype testing

‣ Competitive analysis

‣ Market analysis

‣ Stakeholder interviews

‣ Usability Tests

‣ Analytics Review

‣ A / B Tests

‣ Heat mapping

‣ Eye tracking

‣ Guided exploration

‣ Unguided exploration

‣ Surveys


THE INTERVIEW

THE FINDINGS

Q&A



TYPES OF RESEARCH‣ Ethnographic research

‣ Stakeholder interviews

‣ Competitive analysis

‣ Usability testing

‣ Usage analytics

12



THE INTERVIEW

THE FINDINGS

Q&A



ETHNOGRAPHIC RESEARCH‣ Understand people in the context of their normal environments

‣ Observe how digital and physical factors co-create an experience

‣ More exploratory in nature; focused on the why of behavior

‣ Conducted in the field (at home or office)

‣ Conducted on the individual’s own devices

13



THE INTERVIEW

THE FINDINGS

Q&A



STAKEHOLDER INTERVIEWS‣ Shed light on beliefs which influence company thinking

‣ Pinpoint where teams disagree internally

‣ Understand goals and priorities of each department / stakeholder

‣ Provide insight into who will be best collaborators

‣ Identify what’s been tried in the past; what worked; what didn’t work

‣ Understand what internal biases teams have

‣ Figure out what other products are influence int he teams; what are their “mentor sites”

14



THE INTERVIEW

THE FINDINGS

Q&A



COMPETITIVE ANALYSIS‣ Assess common themes within the industry

‣ Understand table-stakes features users would expect as the bar of the experience

‣ Identify opportunities that are not being met in the industry

‣ Acknowledge new features that may require user education / behavior change

15



THE INTERVIEW

THE FINDINGS

Q&A



USABILITY ANALYSIS‣ Understand why people do what they do within the product

‣ Insight into motivations, fears, goals of the customer

‣ Identify how site messaging & brand impacts usage

‣ Identify how site performance impacts brand & usage

‣ Various levels of formality (lab tests vs casual conversations)

16



THE INTERVIEW

THE FINDINGS

Q&A



USAGE ANALYTICS‣ Understand what people are doing

‣ Quantitative analysis of how people use the product

‣ Identify behavior patterns

‣ Visualize common navigation paths

‣ Understand conversion funnel effectiveness

‣ Observe how changes to the product affect KPI’s over time

17



THE INTERVIEW

THE FINDINGS

Q&A



CHOOSING THE RIGHT RESEARCH FOR YOUR GOALS‣ Different types of research lend themselves to different goals, budgets, & timelines

‣ Products not yet in market require more quantitative research, unearthing, hypothesis validation

‣ Products already in market require more qualitative research, analysis, and guided tests

‣ If there is internal friction; research can help bring focus back to the actual user

‣ Helps internal departments operate on a common set of assumptions & hypothesis

18



THE INTERVIEW

THE FINDINGS

Q&A



SCREENERS: IDENTIFYING WHO YOU SHOULD TO TALK TO‣ Designed to identify people who meet your criteria

‣ Incentivize them to participate

‣ Complexity of screener depends on type of recruiting you do

‣ Quality of your recruits depends on quality of your screener

‣ Don’t underestimate the time to create a good screener

19



THE INTERVIEW

THE FINDINGS

Q&A



ANATOMY OF A SCREENER‣ Very formulaic and linear

‣ Series of yes / now and multiple choice questions with response logic

‣ Identify which unmet criteria result in termination

‣ Acceptance criteria indicated to screener

20

USABILITY RESEARCH WORKSHOP INTRODUCTION

CRITERIA

INVITATION TO PARTICIPATE If the person did not meet all the criteria. Confirm interview time, date, & incentive.

TERMINATION If the person did not meet all the criteria.

question

question

question

question


THE INTERVIEW

THE FINDINGS

Q&A



WEEDING PEOPLE OUT / DISQUALIFYING FACTORS‣ Similar to jury selection; goal is to eliminate outliers

‣ Industry trends: does person use relevant products?

‣ Outlier factor: does person have extreme views on privacy, etc?

‣ Insider factor: is person too close to the industry or product?

‣ Bitter experience: is person tainted from a previous bad experience?

‣ Ability to communicate: can they speak their ind, will they be a good talker?

‣ Terminating with grace: If someone doesn’t meet criteria, end screener eloquently

21



THE INTERVIEW

THE FINDINGS

Q&A



TYPES OF QUESTIONS FOR A SCREENER

22


Biographical Digital Behavior Industry & Product

‣ Age

‣ Gender

‣ Income level

‣ Lifestyle

‣ Education

‣ Type of job

‣ Location

‣ Hours online

‣ Usage of social media

‣ Usage of major sites

‣ Type of internet provider

‣ Familiarity of major sites

‣ Familiarity of topic

‣ Importance of topic in their life

‣ Familiarity of product / brand

‣ User type (new, potential, loyal, etc


THE INTERVIEW

THE FINDINGS

Q&A



SNAPSHOT OF A SCREENER

23



THE INTERVIEW

THE FINDINGS

Q&A



RECRUITING: HOW TO FIND PARTICIPANTSTraditional

‣ Professional recruiting firms

‣ In house recruiting teams

24


Non-traditional

‣ Friends, family, social networks

‣ Craigslist

‣ Existing users who you have relationships with

‣ New visitors who you reach out to


THE INTERVIEW

THE FINDINGS

Q&A

PART 2

The Interview: Plan, Process, & Practice



WHAT WE’LL COVER‣ Planning your discussions

‣ Tips for creating a discussion guide

‣ Why tone matters

‣ Observing body language and behavior

‣ An example interview

‣ What not to do

‣ Logistics tips

26



THE INTERVIEW

THE FINDINGS

Q&A



PLAN: THE DISCUSSION GUIDE‣ Detailed outline of what will happen in the interview

‣ Designed to ensure all questions are captured

‣ Tool to confirm stakeholder needs are being met

‣ Create a conversation with participant

‣ Not intended to be a script; don’t need to ask in order

‣ Familiarize, but don’t memorize questions

‣ Flexible & iterative; could adjust from participant to participant

27



THE INTERVIEW

THE FINDINGS

Q&A



ANATOMY OF A DISCUSSION GUIDE‣ Participant background and understanding

their day to day life; technology basics; and the role the Internet plays in their life

‣ Experience and knowledge of the industry and product at hand (if product is being revealed to them).

‣ Guided tests; having user try and do specific pre-defined tasks while being observed and talking through why they make the choices they make

28

USABILITY RESEARCH WORKSHOP INTRODUCTION

Thanks, preamble, length, topic.

PARTICIPANT BACKGROUND - Day to day lifestyle - Technology basics - Social web presence

SUMMARY & CLOSE Thank them, let them know any follow up and when they can expect payment.

MAIN TOPIC & INDUSTRY - Industry knowledge / interest - Product explanation - Guided exploration (brand, trust, etc) - Unguided exploration (tasks & tests)


THE INTERVIEW

THE FINDINGS

Q&A



DISCUSSION GUIDE: BEHAVIOR QUESTIONS

29



THE INTERVIEW

THE FINDINGS

Q&A



DISCUSSION GUIDE: PRODUCT USAGE

30



THE INTERVIEW

THE FINDINGS

Q&A



GUIDED & UNGUIDED PRODUCT EXPLORATIONGuided

‣ Direct user to specific parts of product

‣ Insight into trust, perception, value, rand

Unguided

‣ Pre-determined scenarios & tests

‣ Designed to test key user flows, screens, and interactions

‣ Provide key anecdotes for findings & reports

‣ Importance of doing these tests yourself beforehand 31



THE INTERVIEW

THE FINDINGS

Q&A



THE INTERVIEW: TIPS & TECHNIQUES

32


‣ Ask open ended questions that get people to tell their stories

‣ At all costs, avoid yes / no questions

‣ Try not to suggest answers for the participant; don’t lead them

‣ Don’t be afraid of silence or pauses (it’s their thinking time)

‣ Get people to elaborate by following up with one of the 5 W’s

‣ Play dumb; get the participant to educate you

‣ Paraphrase what you heard

‣ Be aware of your body language (lean in, nod, etc)

‣ Practice mirroring to make them feel comfortable


THE INTERVIEW

THE FINDINGS

Q&A



DEMO INTERVIEW: 1 ON 1

33



THE INTERVIEW

THE FINDINGS

Q&A

Have respondent introduce himself / herself to get comfortable and share basics. ‣ Describe a week in the life: what are the main activities, how do you spend your time,

what are the things you’re planning for in the future ‣ What are your personal hobbies or interests? ‣ How much would you say you use the internet? Where do you normally access it? ‣ What do you do when you first sit down to your computer? When you first open your

browser? ‣ Are there specific websites that you use all the time? For fun? For work? Which do you

like the best and why? How did you learn about these sites? ‣ Do you subscribe to any pay services online? (premium content, tools, games, music,

etc.) Why or why not? ‣ When and how do you tend to use your devices? ‣ What applications do you use most often on your phone and/or tablet device? What are

the 3 applications you use the most? ‣ What is the latest application you downloaded? ‣ Did you pay for any applications? How much did you pay? Did it bother you to have to

pay for them?



INTERVIEW: LOGISTICS TIPS

34


‣ Have two people (1 male / 1 female) conduct the interview

‣ Record the interview (with permission)

‣ Consider noise level when choosing location for interview

‣ Have guide on an iPad or laptop (papers = loud / fidgeting)

‣ Have one person take notes so they can record non verbal findings (mouse & eye movement)

‣ Take their photo (with permission)

‣ Minimize distractions (turn off devices, don’t be hungry, etc)

‣ Allow for transit time / directions ahead of time

‣ Setting expectations / dealing with no shows


THE INTERVIEW

THE FINDINGS

Q&A

PART 3

Team Creation: Teams and Planning



Teams and planning

36


‣ Break into 4 person teams

‣ Start planning out the research

‣ Create sets of partners to conduct interviews

‣ Choose which set of partners will interview which hotels

‣ Plan your schedules for each destination

‣ Limit your interviews to 45 minutes max

PART 4

Let’s RESEARCH

Matt: +1 808 721 5050

Dan: +1 917 553 4396

PART 5

Feedback: Day 1 learnings and questions

PART 6

The Findings: Drawing Conclusions and Taking Action




40


9:30am - 10:30am

10:30am - 2:30pm

2:45pm - 4:00pm

4:00pm - 4:30pm

4:30pm - 5:00pm

Part 6: Findings: Drawing Conclusions & Taking Action

Part 7: Team Analysis

Part 8: Presentations

Part 9: Feedback and Outcomes

Part 10: Q & A


THE INTERVIEW

THE FINDINGS

Q&A



WHAT WE’LL COVER

41



THE INTERVIEW

THE FINDINGS

Q&A

‣ How to establish your findings

‣ Mapping findings to themes

‣ Supporting findings with quantitative data

‣ Tips for presenting your findings

‣ What to avoid



WHEN TO START CONNECTING THE DOTS

42


• Identifying themes

• Schools of thought

• Between each interview (conversation what we heard)

• Key standout quotes

• Adjust discussion guide

• Patterns after x interviews (adjust guide w/ not agenda)

• Rough headlines for presentation deck

• Don’t worry about being wrong (contradictions, etc)

• Findings don’t always have to be opportunities; could be dissonance


THE INTERVIEW

THE FINDINGS

Q&A



FINDINGS: WHAT WE HEARD

43


• Remind people of the goals

• Recap participant criteria, location, number, length

• Recap research methods & techniques

• Recap who conducted the research


THE INTERVIEW

THE FINDINGS

Q&A



FINDINGS: WHAT WE LEARNED

44


‣ Identify themes, observations, and learnings

‣ Have a definitive point of view (e.g.. no vague statements)

‣ Break findings up into key sections:

‣ General user lifestyle, industry, & technology habits

‣ Industry specifics

‣ Product / website specifics

‣ Include actual quotes from participants

‣ Include screenshots to illustrate UX problem areas


THE INTERVIEW

THE FINDINGS

Q&A



FINDINGS: RECOMMENDATIONS

45


‣ Include specific suggestions for quick win changes

‣ Identify parts of the experience where A / B testing could be done

‣ Identify key screens and user-flows to prioritize

‣ Identify follow up research suggestions


THE INTERVIEW

THE FINDINGS

Q&A



WHAT DO FINDINGS LOOK LIKE?

46



THE INTERVIEW

THE FINDINGS

Q&A




47



THE INTERVIEW

THE FINDINGS

Q&A




48



THE INTERVIEW

THE FINDINGS

Q&A

PART 7

Team Analysis: 10:30 - 14:30

PART 8

Presentations

PART 9

Feedback and Outcomes

PART 10

Q&A



RESOURCES & BOOKS

53


Books

‣ Interviewing Users

‣ Mental Models

‣ Just Enough Research

Software & Tools

‣ CrazyEgg

‣ ChartBeat

‣ UserTesting.com

‣ Ethn.io

‣ Intercom.io

‣ Pear Note


THE INTERVIEW

THE FINDINGS

Q&A

http://rosenfeldmedia.com/books/interviewing-users/

http://rosenfeldmedia.com/books/mental-models/

http://www.abookapart.com/products/just-enough-research

THANK YOU.PLEASE STAY IN TOUCH:

Matt Lee Dan Maccarone@DanMaccarone@MLeeConsulting

[email protected] [email protected]

mailto:[email protected]

mailto:[email protected]?subject=

ux poland user research workshop 04.21.15

Internet

right research

uxuser research

user research matters

usability research workshop9

express consent

findings qathis material

comthis material

user researchthis material