usability week 2014 - droppdf1.droppdf.com/files/woqyo/usabilitytesting-pwd-gr8ux-unlocked.pdf ·...

Usability Week 2014

Usability TestingKara Pernice

Nielsen Norman Group

#nnguw

Password: gr8UX

No video recording, audio recording, or photography of this training content is allowed.

Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.

2

Kara PerniceManaging Director, Nielsen Norman Group

Kara has led hundreds of intercontinental research

studies, and generated many design guidelines and useful

and usable designs as a result. She coauthored the

book Eyetracking Web Usability as well as many of NN/g’s

research reports.

Kara is a leading authority on intranet usability and

eyetracking usability (The Wall Street Journal called her

“an intranet guru.”) She judged the submissions for and

coauthored NN/g’s Intranet Design Annuals 2001-present.

She has worked with clients in just about every industry, helping them meet design and strategic challenges.

Kara has more than 20 years of experience in UX design and research for applications,

websites, and other consumer products. She is expert at managing the UX process and

established successful UX programs at Lotus Development, Iris Associates (an IBM

subsidiary), and Interleaf.

She chaired the UX Professionals’ Association 2000 and 2001 conferences, defined the

organization’s submission review process when she chaired the presentation committee

in 1999, and served as 2002 conference advisor. She holds an M.B.A. from Northeastern

University and a B.A. from Simmons College.


3

Topics

• Overview of usability testing

• Planning what to test (and where)

• Writing tasks

• Write tasks for afternoon study

• Facilitating

• Run studies in groups

• Note taking & analysis

• Presenting findings


4

“What people say, what people do, and what they say

they do are entirely different things.” - Margaret Meade


5

Bullet Points vs. Video: Which one Had a Higher Conversion Rate?

www.whichtestwon.com/google-website-optimizer-test-results

Bullets (Version A) had

30% more conversions

than video (Version B)

Version A

Version B


6

Observe Users to Understand Behavior

Interviews are different from

observation because:

• Human memory is fallible

• Users are pragmatic and concrete

www.nngroup.com/articles/interviewing-users

How would

you use this

website?


7

Basics of Usability Testing

• Involves a design, a user and a

facilitator

• One user tests at a time

• Think-aloud methodology

• Observe user behavior

• Listen to user feedback

• Facilitator stays quiet

• Gathers first-hand data from real

users

• Reduce guessing and arguing

• Can be quick, easy and

inexpensive

• Is mainly qualitative


8

Typical Usability Testing is Qualitative

Qualitative Quantitative

Users Few (4-20) Many ( > 20)

MethodObserve behaviors

and draw insights

Run a controlled

experiment

Statistical

validityCannot be

establishedDerived from data

Goal Iterative designDefinitive,

scientific “proof”


9

Testing With 5 Users Uncovers 85% of the Issues

See Appendix: A mathematical model of the finding of usability problems by Jakob Nielsen and Thomas Landauer

User 1

User 2

User 3

All the usability problems in the interface


10

Best Results from Iterative Testing & Design

Budget allocation for usability studies: 20 users

5 5 5 5


11

Qual

Small

sample

(5-10)

Backup insights with numbers.

Good if you need more support.

Quant

Large

sample

(15+)

Shows that there is an issue.

Use qualitative study find out

why


12

We can Measure with Usability Testing

• Learnability

• How fast can new users understand it?

• Efficiency of Use

• How fast can users accomplish tasks?

• Memorability

• Can people use it without having to re-learn it?

• Errors

• Where and why do people encounter problems?

• Subjective Satisfaction (through questionnaire)

• How much do users like using the interface?


13

You can (and Should) Test Now

Gathering requirements

• Field studies

• Usability studies: Competitive analysis

Early iterations

• Usability studies: Paper prototypes

• Card sorting

Middle iterations

• Usability studies: Clickable prototypes

Later iterations

• Usability studies: “Final” design


14

Steps in a Usability Study

1) Determine focus of study

2) Recruit participants

3) Write tasks

4) Determine location and set-up

5) Conduct the test sessions

6) Analyze findings

7) Communicate findings (report and/or presentation)

8) Improve design

Involve team members throughout the process


15

Deciding What to Test

• Overall usability of whole system

• Top tasks

• Focus on certain user group

• Novice users, domain experts, seniors, etc.

• Limit scope to specific areas or features

• Important to the business

• Competitive tasks

• New feature

• Changed feature

• Area you know causes problems

• Realistic scope for each study


16

Overview of QualitativeUsability Testing Options

In-person

(facilitator and participant

in same location)

Remote


can be anywhere)

Moderated

(facilitator “talks” to

participant while

conducting tasks)

Traditional one-on-one

user testing

One-on-one user testing

via online screen sharing,

such as GoToMeeting

Automated

(participants conduct tasks

on their own)

Somewhat pointless,

although allows you to run

sessions simultaneously

Setup study in online tool

such as usertesting.com or

whatusersdo.com


http://www.gotomeeting.com/online/

http://www.usertesting.com/

http://whatusersdo.com/

17

Many QuantitativeRemote Automated Options

In-person


in same location)

Remote


can be anywhere)

Moderated

(facilitator “talks” to

participant while

conducting tasks)

Traditional one-on-one

user testing

One-on-one user testing

via online screen

sharing, such as

GoToMeeting

Automated

(participants conduct

tasks on their own)

Somewhat pointless,

does allow you to run

sessions in parallel

Benchmarking, click

testing, card sorting, tree

testing, prominence and

recall, performance,


http://www.gotomeeting.com/online/

18

LogisticsFinding participants and setting up a “lab”


19

Keep Track in a Test Plan

• Objectives of the study

• Location and date of test

• Version or section being tested

• Description of target users

• Tasks and goal of each task

• Metrics and how they’ll be measured

• If (or when) the facilitator will be allowed to help the user

• State of system at start of each task and session

• What to do between tests

• List of needed materials

Example of test plan in Appendix


20

Recruiting Options

Pros Cons

In-house • Direct control over recruiting

• No recruiting fees

• Time and labor intensive

• Difficult if there is no

established ‘user pool’

Outsourcing • Frees team to work on other

aspects of testing

• Hired company has resources

for and experience with

recruiting

• Cost

• Reliant on quality of

recruiting firm and screener


21

Finding Participants

• Actual customers

• Customer lists

• Advertisements

• Friends and family referrals

• Colleagues

• Website

• Outside organizations, clubs

• Recruiting agencies

• Online panels

• Research Now, Schlesinger Online, Toluna, SSI

• Site intercepts

• Ads: Twitter, Craigslist, LinkedIn


http://www.researchnow.com/en-GB/Panels.aspx

http://www.schlesingerassociates.com/online_solutions.aspx

http://www.toluna-group.com/

http://www.surveysampling.com/

22

Recruiting Participants for Remote Studies

• Easiest to use provided panel

• Use basic demographics criteria and supplement with self-selection

questions

• Be very cautious of “professional testers”

• Filter out “cheaters” by looking at time-on-task and open-ended

feedback

• Always recruit more participants than you need


23


24

ethnio: Recruit Users on Your Site Now

Recruit for later time pointHave them continue using the site as they were

Recommended book: Remote Research: Real Users, Real Time, Real Research (2010), by Bolt and

Tulathimutte


http://www.amazon.com/Remote-Research-Real-Users-Time/dp/1933820772/ref=sr_1_1?ie=UTF8&qid=1397068214&sr=8-1&keywords=remote+research

25

Requires 2 Lines of Code in <body>Copyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.

26

Sample ScreenerCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.

27

ethnio branching logic

• Send to unmoderated study

• Call to start moderated study

To unmoderated study: http://skinnywhitegirl.com/blog/ethnio-usertesting-com-for-real-time-unmoderated-usability-testing-goodness/521/To moderated study: http://www.nngroup.com/articles/live-intercept-remote-test/


http://skinnywhitegirl.com/blog/ethnio-usertesting-com-for-real-time-unmoderated-usability-testing-goodness/521/

http://www.nngroup.com/articles/live-intercept-remote-test/

28

Keep Intercepted Participants on the Phone

1. Call participant at phone number provided during recruiting

• For international participants use Skype

2. Instruct them how to share screen

3. Start 3-way call with number from screen sharing software

• Tell participant: “Before we can start I need to conference in the

audio line. I'm going to place you on a brief hold. If for some

reason we get disconnected I will call you right back.”

4. Use integrated audio recording

• Make sure you're not muted or the audio won't record


29

Sample Live Intercept “Task” Instructions

Please continue to use the website as

you were before. Finish anything you

were working on that you didn't get to

you.

Remember to tell me what you're

thinking as you're working.


30http://help.ethn.io/hc/en-us/articles/200202485-Pageviews-Screener-Views-and-Responses-the-Funnel-


31

Consent Form and/or NDA (if necessary)

• Create online consent form

• Tool suggestions:

short.nngroup.com/UXTools

• Email participants ahead of

time

“In order to participate, you

need to read and sign the

consent form. This allows me

to record the GoToMeeting

session. Please do this ahead

of time.”

• Send them link at the

beginning of session



https://nng.wufoo.com/forms/nielsen-norman-group-consent-to-record/

32

Paying IncentiveCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.

33

Participant Incentives

• Outside Participants

• “Cash” gift certificates (e.g. Amazon or American Express)

• Products or software the company makes

• Cash ~$100 USD/hour

• Amount varies depending on location, profession, length of session

• General consumer or students $75 USD for 90 min

• High-end professional $250 USD for 90 min

• Internal Participants

• Sometimes just a heartfelt “thank you” email

• Gift certificates, cafeteria coupons

• T-shirts, mugs, etc.


34

Response and Abandonment Rates

• Screening response rate depends on:

• Incentive amount

• Perceived time needed

• Number of questions

• Wording of questions

• Screening criteria

• Abandonment (study completion):

• Offer high enough incentive

• Only pay participants who complete study

• Have at least 1 phone call so participants are more committed (esp.

important for qualitative studies where you rely on feedback)

Average survey

response rate

is 10%


35

NN/g Case Study: Tree Test on Intranet

RESPONSE RATE 56%

• 550 e-mail invitations were sent and 306 people participated

ABANDONMENT RATE 35%

• 204 (66%) of the 306 respondents completed all 11 tasks

• 102 (34%) people abandoned the study

• No real incentive, but the people work for the organization


36

Cost of Unmoderated Remote Usability Testing

Panel Own participants

Recruiting $50/participant $40/participant

Incentive Incl. with recruitment $25

Prep 3 hrs. 12 hrs.

Watch recordings30 min./participant

(for 20 min. videos)

30 min./participant

(for 20 min. videos)

Analysis 4 hours 4 hours

Total for 5 users $250+ 9.5 hrs. $325+ 18.5 hrs.


37

Compared to in-person testing, savings from not renting and having

to travel to testing facility

Cost of Moderated Remote Usability Testing

Software $15/month

Recruiting $40/participant

Incentive $80/participant

Prep 16 hrs

Running sessions90 min/user (for 60 min

session)

Analysis 8 hrs

Total for 5 users $600 + 31.5 hours


38

Items (and order) for the Screening Script

• This is not a sales call

• Reason for session: get your feedback to help us improve the

product

• One-on-one session, NOT a focus group

• Length, date, and location of session

• Incentives

• Explain any videotaping or audio taping

• If interested and available, go through screening questions

Example of screener in Appendix


39

Study Locations

• Your office

• Larger team can watch

• Allows you to test more participants per day

• Easy room/ system set-up

• User’s home or office

• Provides context

• Accommodates user’s schedule

• Easier for accessibility studies

• Usability lab or location other than your office

• Use for competitive studies

• Comprehensive system and observation set-up


40www.humancentric.com/#/labs/cary/facility-information


41

Turn Any Room into a “Lab”

• User in an office/conference room

• Usually four or fewer observers in the user room

• Extra observers in a nearby conference room

• Computer monitor or TV monitor via coaxial cable

• Speaker phone


42


43


44

Conference room

External monitor

Testing computer

Participant

Facilitator

Observers


45


46

Basic Remote Testing Setup

• Facilitator and participant are not in the same physical location


47

Remote Can Include Observers

• Facilitator, observers, and user are in different locations


48

When to Move Out of “The Lab”

• See users’ own computer setup

• Especially if using a highly customized system

• Test with people currently using site

• Test with people around the world

• Easier to convince people to participate

• Involve observers in multiple locations

• On tight timeline

• (Cost)


49

Folders in

bookmark

bar

5 tabs open


50

Participants’ Locations for Recent ProjectOnline Remote Study

Facilitator’s location


51

Remote can be Great for Testing Early Designs

• Create online clickable prototype

• e.g., Balsamiq, HotGloo, Solidify

• More options: short.nngroup.com/UXTools

• Link to hosted prototype instead of website

More on testing prototypes with usertesting.com: http://blogs.balsamiq.com/product/2013/05/15/usertestingcom/



http://blogs.balsamiq.com/product/2013/05/15/usertestingcom/

52

When to Conduct In-Person Studies

• When it needs to seem more immediate and “real” to the team

• See people’s faces and body language

• When you don’t want your design or info. to be subject to

screenshots or photos (secure, private control who sees the design)

• Manage the situation and tasks more easily

• More personal, sometimes more comfortable for all

• Very personal tasks with sensitive data

• Poor internet connection

• Less computer-savvy users

• Non-computer based tasks

• Kids (requires consent)


53

In These Remote Studies We Missed What People Were Writing Down Off the Screen


54

Moderated: Facilitator has Direct Contact with the Participant

• Moderated = synchronous


55

Automated: Participants conduct tasks on own, data analyzed later

• Automated = asynchronous = unmoderated

Researcher sets up

study

Researcher analyzes

data

Participants perform

test through online tool


56

Benefits Automated vs. Moderated Remote

Automated Moderated

Fit participants’ schedule

(easier recruiting)X

Automatically capture metrics

such as success and timeX (X)

Ask follow up questions (X) X

Prompt participant for

clarificationX

Change (order of) tasks X


57

Tools for Screen-Sharing and Recording

• Computer-based recording products

• Screen & audio capturing software

• Picture-in-Picture capabilities

• TechSmith: Morae (PC only) or Camtasia (PC and Mac)

• Silverback (Mac only)

• GoToMeeting

• Mobile

• Document camera

• Webcam pointed down at desk

• Mobile device camera

• Screen mirroring app (e.g. Reflector for iOS)

• Screen recording app


58

Moderated Remote Testing

• Benefits of moderated remote testing

• Observe users on their own machine

• Test users in any location

• Ask follow up questions and prompt during session

• Disadvantages

• Can’t always get person’s physical reactions

• Logistics (setting up software on user’s computer)

• Typical set-up

• 2 computers (1 user computer + 1 observer computer)

• Web conferencing software (e.g. GoToMeeting)

• Speaker phones or (headset) mic


59

Unmoderated Remote Testing

• Good for very focused studies with general audience

• No facilitator means you can’t

• ask follow-up questions

• prompt participant for clarification

• ensure the task was understood

• Spend even more time on task writing

• Use existing remote testing tool


60

Writing Tasks


61

Types of Tasks

• Exploratory tasks

• Open-ended, research-oriented

• Examples:

• See if you would invest in this

company.

• Find a mobile phone plan for

yourself.

• Directed tasks

• Specific, answer-oriented

• Examples:

• Find the contact information for the

PR department.

• How much does it cost to buy the

product?

• Discoverability

• Don’t tell about a feature

• Give a small hint for the need for

the feature

• Findability

• Tell the feature exists (not exact

name) but see if they can find it

• Usability

• Once they are using, what works,

does not work, and why?

• Usefulness

• Feature seem helpful, interesting,

needed (sometimes better to

judge this using other methods)


62

Number of Tasks

• Prioritize tasks

• Give essential tasks first

• Prepare additional tasks in case of extra time

• Estimate task time

• Pilot studies will help you with this

• Consider time limit

• After 5 min. say: “I’m going to stop you there. Thank you for doing that

activity.”

• Is the priority to get through many tasks, or hit a select few deeply?


63

Exercise: Task Critique

• Review the tasks in relation to the task goal

• Decide if they are appropriate

• If not, identify the problem with the task and re-write it


64

Task-Writing Reminders

Consider the goal of the task

Make it realistic

Give minimal context (not overly scenario-based)

Don’t try to be funny

Keep language neutral, unbiased, and understandable

Leave out clues or hints Avoid having micro-steps

Involve team in writing and/or reviewing tasks

Have a clear call-to-action

See Appendix: Criteria for Evaluating Usability Tasks


65

Task Logistics

• Call them “activities”, not “tasks”

• Try to make the first task an easy one

• Write each task on a separate sheet

• Tip: Do not write “task” or “1” at the top of the task page. Instead use a

small page number as your indicator.

• Ask participants to think out loud and tell you when they are finished

Participant is

reading the task


66

In an unmoderated, remote study, the tasks become far more important

• The facilitator cannot fix things during the session

• Tell the user he can stop

• Clarify a task

• Answer and non-UI-related question

• Ask the user to do the task, not just talk about it


67

Types of Research Questions for Remote Studies

Appropriate research questions for this method:

• Is the goal of this page clear?

• Do people know what they can do with this app?

• Does this feature or site area work as we intended?

• Can user accomplish this specific task?

• Can people find what they’re looking for?


68

In Remote Studies: Focus, focus, focus

• Session time typically limited to 15-20 minutes

• Select a few specific elements, rather than full overview of site

• Good starting point is top tasks that all users must be able to

accomplish in order for product to be a success

• Test things that you’re not sure about

• Test unconventional interaction models


69

Problem: Participant doesn’t interact with the feature or visit the page you had intended

• Be explicit, tell them to use feature or go to a certain page

• See if the TSA blog (http://blog.tsa.gov) gives any helpful advice for

traveling over the holidays. Move on to task 3 when you feel ready or

if 5 minutes have passed.

• Ask for comments on specific element in follow up question

• If you didn't visit the following page in Task 2, please use it now to

see if the TSA gives any helpful advice for traveling over the

holidays. http://1.usa.gov/1pqySss


http://blog.tsa.gov/

http://1.usa.gov/1pqySss

70

Problem: Participant doesn’t know when to stop the task

• Use directed tasks with a specific correct answer

• Set time-limit on open-ended tasks

• Use the internet to find a drycleaner near you. Stop when you have

found one or if 5 minutes has passed.

• …Move on to task 2 when you feel ready or if 5 minutes have

passed.


71

Problem: Participant selects the first product or service they notice, rather than the one that’s best for them

• Be specific in task with clear goals

• Go to www.houzz.com. Use the site to find a coffee table for your

home. You want to spend less than $200.

• Rent a car for 4 people and luggage from July 10-July 14 at the

Amsterdam airport.


http://www.houzz.com/

72

Problem: Participant just reads answer from interface (or memory), rather than trying to understand content

• Ask them to rephrase or relate it to them

• See if you’d be interested in signing up for this service. Give at least 2

reasons why or why not.

• Use the website to find out what projects the World Wildlife

Federation is doing in the Arctic. Describe them in your own words.

• Briefly describe why you might or might not recommend the site to

your friend.


73

Problem: Sensitive information

• Use “Blur screen” feature (in usertesting.com)

• Provide a dummy credit card

• Recruit only people willing to login

• Provide dummy account information


74

Think Aloud Instructions

• Give participant a printed paper to read out loud (see Appendix

p.14)

• Gets them used to hearing their own voice

• Ask them to think aloud while removing staples from a stapler

• Show them video of good thinking aloud


75

Remote Sessions: Provide Instructions

• Encourage participants to think out loud and elaborate on their

experiences

• Explain that some activities might be very short and not to worry

about the number of tasks that are left

• Tell them to limit background noise (e.g. TV, radio, kids)

• Provide brief scenario as context for why they are conducting these

activities


76

Sample Remote Test Instructions

• You will be looking at several different websites. Please think out

loud as you do the activities and stop whenever you normally would.

• Tell me what you’re looking at, why you click a link, and what you

expect to happen.

• Some activities will be very short and others will take longer. Don’t

worry about getting through the activities quickly. Completing all

activities shouldn’t take you more than 20 minutes.

• Before we start, please turn off any background noise.

• Now, imagine a friend recommended this website and sent you a

link via email.


77

Metrics You Might Collect

• Success (scale 0-4 or pass/fail)

• Difficulty (none, some, very)

• Time on task

• Automatically logged or measured with silent stopwatch

• Errors

• Subjective satisfaction

In qualitative study use metrics to locate issues, not extrapolate

to all users


78

Gather Ratings After Each Task

I am confident with the answer I gave.

Not at all Very

1 2 3 4 5 6 7

I found the activity to be easy.

Not at all Very

1 2 3 4 5 6 7

I found the activity to be frustrating.

Very Not at all

1 2 3 4 5 6 7

Please rate the activity you just worked on based on a scale of 1 to 7,

with 7 being the most positive answer.

Circle one number for each question.


79

Remote Tests: Post-Task Questions

• Especially helpful if people didn’t think out loud much

• Post-task questions:

• How easy or difficult was this activity?

• Any other thoughts or comments on working on this activity?

• Difficulty or satisfaction rating

• On a scale from 1-5, how easy or difficult was that activity?


80

Post-Session Follow-Up Questions for Remote

• Usually written (not verbal)

• But I like to give the questions as a task in the system, so it is

recorded and the user can say his answer

• Don’t ask participants to write down how they used the site (you can

go back and watch)

• Focus on overall quality:

• Net promoter score

• Satisfaction rating

• “Would you come back on your own and visit this website or use this

app?”


http://www.netpromoter.com/why-net-promoter/know/

81

Written Responses VaryCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.

82

Exercise: Plan Study and Write Tasks

• Form groups of 3-4 people

• You’re UX specialists on a team creating a new mobile calendar

application. Before you design the prototype you want to know how

well the previous version works.

• Volunteer one mobile phone or tablet per group

• Determine what you want to test. What questions you have?

• Most common, critical, features? (e.g. adding appt, looking up date)

• Integration with other applications?

• Write 2-3 tasks

• Include exploratory and directed tasks

• Where should participants start the session?


83

Conducting the Sessions


84

Typical Schedule

• 60 - 90 minutes per session

• 4 - 5 users per day

• Example of testing day:

• 9am – 10am

• 10.30am – 11.30am

• Lunch

• 12.30pm – 1.30pm

• 2pm - 3pm

• 3.30pm – 4.30pm

For rapid, iterative prototype testing, add more time between sessions

so you can change the design between users


85

Session Steps

1) Set up

2) Prepare observers

3) Welcome participant

4) Pay incentive

5) Sign consent form

6) Run test session

7) Debrief user

8) Debrief observers

9) Prepare for next session (gather materials; reset the app, i.e.

clear cache in the web browser)


86

Include Observers

• Don’t test in isolation

• Invite everyone

• Encourage them to come to at least 3 sessions

• Encourage people to take notes

• Provide tips on what to look for

• Provide printed screenshots of design to take notes on

• Ask them to at least not top 3 usability issues for each participant

• Allow them to ask the participant questions (through facilitator)

• Make sure they stay silent if in the same room

• Have a brief discussion after each test session


87

Prepare Participant When He Arrives

• Make her feel comfortable

• Introduce observers in in the same room (no job titles or info

usually)

• Acknowledge the unnatural situation

• “We are evaluating the product, not you…”

• “Your feedback, both good and bad, will help us improve the product”

• Explain lab set-up and test procedure

• Tell them you will be quiet most of the time

• Offer drinks and breaks

• Sign forms, give incentives

Example of facilitation script in Appendix


88

While Facilitating

• Monitor

• Tasks

• Session time

• User comfort

• System

• Observers

• Take notes

• Consider whether to

interrupt or ask questions


89

How to Communicate with the Participant

• Give subtle acknowledgements

• Uh-huh, OK, nodding

• Refrain from being interruptive or chatty

• Stay neutral

• Don’t ask leading questions

• Avoid prompting

• Avoid explaining or defending interface

• Avoid answering questions or helping out too early

• Get clarification

• Echoing technique

• Trailing-off technique

http://www.nngroup.com/articles/talking-to-users/


90

Helpful Phrases to Recall When Facilitating

• “What would you normally do?”

• “What do you think?”

• “Any (other) thoughts or comments?”

• “Remember to tell me what you’re thinking.”

• “Tell me more about…”


91

Exercise: Facilitation and Observation

Part 1: Preparation for full usability session (10 min)

• Break into same groups of 3-4

• Choose roles: One facilitator, one participant, rest will be observers

• Write each task you wrote on a separate piece of paper

• Decide on scenario to tell participant. Why are they here using this

phone or tablet?

• Facilitator read over (and edit) facilitator script

Part 2: Run usability study (20 min)

• Participants move to another group

• Find quiet area to run study

• Observers take notes on facilitation


92

Discussion: Review Qs a Facilitator Might Ask

• What do you think of the colors and

fonts?

• Did you pay attention to these images?

• Why did you not scroll down to see all of

the options?

• Would you use this feature?

• It seemed like you were having difficulty

in this area. What were you thinking?

• Did you notice this link?

• Did you think it was easy to use this

website?


93

Discussion: Managing Challenging Situations

What do you do if the participant ...

• is uncomfortable having his face recorded

• doesn’t fit the right profile

• is uncomfortable signing the consent form

• starts blaming herself/himself

• asks if he/she is right

• tells off-topic stories or jokes during testing

• is elderly and not comfortable using computers

• is 8 years old and can’t sit still

• talks about what they would do, but doesn’t actually do it


94

Note Taking


95

What to Capture

• Chronological happenings

• User’s path

• Issues and designs that work

• Themes

• Body language

• Quotes

• Assumptions and conclusions

• Write, then file away: Do not redesign while watching the studies

• Questions you want to ask at the end of the session

• Metrics you decided to look for, such as success and time

• TIP: Format a document, spreadsheet or whiteboard with the

questions before the study


96

Note-Taking Tools

• Paper

• Notepad or sticky notes

• Basic text document or spreadsheet

• Usability logging software

• Morae Observer, www.techsmith.com

• http://usablog.willsansbury.com


97

Excel with Automatic Timestamping

Timestamp setup tutorial: www.howtoexcelatexcel.com/excel-tips-tricks/create-a-timestamp-in-excel-with-formulas/


http://www.howtoexcelatexcel.com/excel-tips-tricks/create-a-timestamp-in-excel-with-formulas/

98

Exercise: Observing and Taking NotesCopyright*©*Nielsen*Norman*Group,*All*Rights*Reserved.*Please*do*not*share*electronic*or*printed*copies,*or*post*to*the*web*or*your*corporate*intranet.

99

What to Take Notes About

• What the user is doing

• Problems

• Features that work

• User quotes

• Questions to ask participant

• How easy or difficult it seems to be to complete the task

• Easy

• Some difficulty

• Difficult (or could not complete)


100

Turning Notes into Findings


101

Analysis Can Feel Overwhelming at First

• Review by task or category

• What the user was trying to do

• What the user did, path,

backtracking

• Major themes that emerged

• Study how user reacted

• Emotions or expressions

• Enlist observers’ help

• Use affinity diagramming to make

sense of notes from multiple people

Affinity Diagramming tutorial: www.mindtools.com/pages/article/newTMC_86.htm


102

Use Metrics to Locate Issues

• Success rates

• Which tasks were most (un)successful? Why?

• Time on task

• Are users successful, but slow?

• Have we met any goal times?

• Satisfaction ratings

• What features/areas did users (dis)like?

• Error rate

• How many participants ran into the same issues?


103

Common Mistakes

• Presenting recommendations as though they are findings

• “The login should be moved from within the banner.”

• Treating five users like a statistical sampling

• “50% of users didn’t see the login area”

• Better: All, some, most, none, only one out of the five

• Blaming the user for the interface error

• “Participants didn’t understand why they had to log in.”


104

Make Recommendations

• Provide suggested improvements to the design or next steps

• Base recommendations on results from testing or best practices

• Don’t make visual design recommendations if you’re not the designer


105

Examples Finding with Recommendation

HIGH: The benefits of creating an account weren’t clear, so

users were annoyed when they were asked to do so.

Ask for email address as username, so that users only have to

create (and remember) a password.

List out of the benefits of creating an online account.

MEDIUM: The button to invoke search is a tiny, difficult to click,

button.

Replace the small icon with a larger button, named Search.


106

Determine Severity of Each Finding

GOOD The design was simple and participants understood it.

HIGH Participants could not complete a task or experienced

significant frustration.

MEDIUM Participants had some difficulties completing a task, but

could complete it.

LOW Participants experienced some minor frustration,

confusion, or other issues.


107

Determine Severity of Negative Findings

1. Number of participants who encountered the problem

2. Frustration caused by the problem

3. Difficulty to recover from the problem

4. The same user encounters the problem more than once

5. Problem occurs on critical tasks or in critical area of the site

1 2 3

Most Some Few

1 2 3

Extreme Some None/Little

1 2 3

Extreme Some None/Little

1 2 3

Most Some Few

1 2 3

Always Sometimes Never


108

Scoring Severity of Negative Findings

• Add totals from previous slide

• Score

• HIGH: 5-8 points

• MEDIUM: 8-12 points (make own judgment call at 8 and 12 points)

• LOW: 12-15 points

Severity Rating Tool (Excel) on USB drive


109

Exercise: Analyze Findings

• Break into same groups

• Review and discuss notes

• Write 2-3 findings

• Include at least one good thing

• Determine severity of each

• (Extra credit: Write a recommendation for fix)


110

Reporting Usability FindingsReports can be text, spreadsheet, or presentation style


111

Purpose of Report

• Make findings understandable and actionable

• Present issues and usable designs in a simple format

• List only necessary and important information

• Have report available in a timely manner

• Share usability information with many teams

• Save information over time (history document)

• Share improvements


112

Purpose and Schedule Should Dictate the Report You Create

• Deliver in an hour, up to 24

hours

• Bulleted list of top findings

• Short description of

problems and usable

designs

• Usually 1 to 4 pages long

• Deliver in one to two

weeks

• In-depth analysis

• Includes screenshots and

annotated call-outs

• Anywhere from 10 to 100

pages

• Might do this for a project

ending or competitive

study

Informal Formal

Example of formal written usability report in Appendix


113

Presenting Findings

• Involve and acknowledge all team members

• Ask for their input

• Foster inclusiveness in presentation

• Don’t have surprises

• Briefly describe the test

• Talk about positive findings and areas for improvement

• Avoid opinions, stick to facts

• Use creative visualization to make data memorable

• Annotated screenshots, videos, photos, charts & graphs


114

“This must be the 10th time they said this, ‘treats both main causes

of asthma symptoms—airway construction and inflammation’. I've

read that numerous times already. It is too repetitive.”


115

Video

• Use it to show behavior and feedback from users

• Especially good for people who couldn’t make it to test sessions

• Edit clips of:

• several users doing the same task

• same feature across releases

• same feature on your site and competitors’

• Can do as a report, but not searchable or easy to use as a historical

tracking tool


116

Wedding registries were difficult to find through search, because radio buttons weren’t noticeable.Several participants searched within wish lists, not registries, because they didn’t notice radio buttons .


120

Tracking Usability Findings


121

Tracking Across Product Releases

• You can’t fix everything

• Track usability issues in a database, e.g.:

• Excel spreadsheet on intranet

• Google doc

• Bug tracking system

• “Usability issues” database

• Keep full reports in a searchable repository


122

Classify the Time to Fix Each Issue

• 3- Difficult (more than X time designer/ more than X time developer)

• 2- Moderate (less than X time designer/ less than X time developer)

• 1- Easy (less than X time designer/ less than X time developer)


123

Example Grid: Add Numbers and Sort

Severity Resources Total Next Steps

Users didn’t know what

information to expect in

the “services” area.

3 – Low 1 – Easy

Des: 15 min

Dev: 15 min

4 Jenny comes

up with new

label. Lisa

implements.

The gray color of the “add

to cart” button made users

think it was not active.

1 – High 2 – Moderate

Des: 2 days

Dev: 1 day

3 Jenny choose

a new color.

Peter

implements.

Focus on low totals: big issues that are easy to fix


124

Summary

• Test now and often

• Involve other team members

• Respect the user

• Don’t talk too much during the test

• Be accurate and fair in your analysis and communication


125

Start Testing Now

•Surveys

•Site feedback

•Benchmark existing site

•User test existing site

•User test competitors

Gathering requirements

•Card sorting

•Tree testing

•Menu labels

Scoping and structuring •Usability test prototypes

•Click testing

•Menu labels

•Prominence & recall

Designing

•Usability testing

•Benchmarking

•Analytics

•Click tracking

•Site feedback

Maintaining


126

Thank you!

#nnguw

@karaann

[email protected]

https://www.linkedin.com/in/karapernice


https://www.linkedin.com/in/karapernice

127

Additional Information


128

Fix and Repeat

Design

Test

Fix

Move to

next fidelity


129

Use an existing online tool

Look for answers to these questions:

Can I write my own tasks?

Will it record audio and user’s screen (video of participant is nice,

but not necessary)?

How long can the sessions be?

Does it support mobile and desktop?

Is there a participant panel or do I recruit my own?

Can I pay my own recruits?

Where are the participants geographically located?

What’s the turn-around?

www.nngroup.com/articles/unmoderated-user-testing-tools

Tool suggestions:




130

Screen sharing tool considerations

Are users required to install application?

Are users required to install plugin?

Will it work through firewall?

Can users share webcam? (for picture-in-picture)

Can you record sessions?

How many people can dial in?

Tool suggestions:




131

1. Please join my meeting.

https://global.gotomeeting.com/join/42

1719813

2. Use your microphone and

speakers (VoIP) - a headset is

recommended. Or, call in using your

telephone.

United States (toll-free): 1 877 309

2070

Access Code: 421-719-813

Audio PIN: Shown after joining the

meeting

Meeting ID: 421-719-813

www.gotomeeting.com


132

Paste GTM

meeting

details Alternatively

use GTM

plugin

Invite

participant


http://support.citrixonline.com/en_US/gotomeeting/all_files/GTM030012

133Screenshot from recording of moderated remote session on nngroup.com using GoToMeeting


134

Alternative: Instead of viewing their screen, have them take control of yours

Pros

• Can test prototypes with dummy data

• Prevents participants from having to

download application you’re testing

Cons

• Prevents you from seeing their browser

and desktop setup

• You can’t take notes on your computer


135

Use built in audio capabilities or phone line

• Generally better audio quality

• Provide toll-free number

• Recommend using headset

(or speaker phone) so they

can type and use mouse

• Recommend headphones to

prevent audio feedback

• Suggest they test their

microphone beforehand

Phone VoIP


136

What to tell participants

• Explain setup and ensure everything’s working

• Explain usability testing and what’s expected of them

• Confirm they’re ok being recorded

• Tell them personal information will remain confidential

• Explain that you will be quiet and want them to think out loud

• Help them share their screen

• Run through how you will provide them with instructions

Sample facilitation script: short.nngroup.com/RemoteFacilitation


http://short.nngroup.com/RemoteFacilitation

137

Instructions and tasks

Chat window

Sample Google Spreadsheet:

short.nngroup.com/RemoteTasks

Link to shared Google Doc


http://short.nngroup.com/RemoteTasks

138

Moderated remote facilitation tips

• Facilitate like you would an in-person study, while:

• Providing more subtle verbal acknowledgements (since participant

can’t see non-verbal cues such as nodding)

• Only prompting if truly necessary (more interruptive than via phone)

• Acknowledging questions, without answering them (or participant

might think call was dropped)

• Be prepared for interruptions


139

Typical steps in online remote session (45-60 min)

Start screen sharing software

(Call participant)

Wait for participant to join

Make sure connection is good

Explain what they’ll be doing

Start recording

Make them presenter

Use chat window for tasks

Ask follow-up questions

End session

Send incentive via email


140

Moderated remote facilitation tips

• Facilitate like you would an in-person study, while:

• Providing more subtle verbal acknowledgements (since participant

can’t see non-verbal cues such as nodding)

• Only prompting if truly necessary (more interruptive than via phone)

• Acknowledging questions, without answering them (or participant

might think call was dropped)

• Be prepared for interruptions


141

Invite people to watch sessions

• Send out link to sessions

• Create separate meeting invite than the one to participants

• Disable EVERYTHING in screen sharing software

• Use separate chat program for questions from observers to be safe

• Acknowledge that there are others on the call


143

Recording sessions

• Built in recording functionality

• GoToMeeting does not record PIP

• Screen and picture-in-picture (PIP) recording

• Use another computer, dialed into online meeting

• Run additional software: Morae or Camtasia


http://www.techsmith.com/morae.html

http://www.techsmith.com/camtasia.html

usability week 2014 - droppdf1.droppdf.com/files/woqyo/usabilitytesting-pwd-gr8ux-unlocked.pdf ·...

Documents