easy & effective usability testing at codemash 2012

E A S Y & E F F E C T I V E U S A B I L I T Y T E S T I N G

P R E S E N T E D B Y C A R O L S M I T H @ C A R O L O G I C

CodeMash 2012January 12, 2012

"The biggest waste of all is building something no one wants"

@ericries #LeanStartupMI via @MelBugai – May 19, 2011

FIND OUT WHAT YOUR USERS REALLY NEED

http://www.flickr.com/photos/ableman/534155207/sizes/o/in/photostream/http://www.flickr.com/photos/ableman/

AVOID MISTAKES

http://www.flickr.com/photos/dwulff/12256075/sizes/m/in/photostream/http://www.flickr.com/photos/dwulff/

Click icon to add picture

MINIMIZE COST & EFFORT

Click icon to add picture

SAVE TIME & MONEY

GET HAPPY, SATISFIED USERS

http://www.flickr.com/photos/cogdog/2194678510/sizes/o/in/photostream/http://www.flickr.com/photos/cogdog/

Create a great, usable, accessible, and relevant experience

WHAT IS USABILITY TESTING?

Steve Krug - http://www.sensible.com/rsme.html

•Real users doing real tasks•Not guided•Using prototypes or live products•Observed in the field

USABIL ITY TEST ING

•Anywhere•Any Stage of dev •Anytime

•Realistic environment

LOCATION ISN ’T IMPORTANT

http://www.flickr.com/photos/nzdave/491411546/sizes/o/in/photostream/http://www.flickr.com/photos/nzdave/

Expert Led• Heuristic reviews• Walk-through• Accessibility review

Required• Usability Testing• RITE Testing• Card Sorting

PARTIC IPANTS

Garrett Goldfield . Fast and Cheap Usability Methods: Using Discount Usability Techniques to Drive Design on Time and Under Budget. http://www.nngroup.com/events/tutorials/guerilla_usability.html

WHY DO USABILITY TESTING?

REGULAR TEST ING

(Yes, this is an old idea; a great one!)

•Team becomes: • accustomed to steady stream of qualitative insight • insight ensures quick decisions

• …line up with business and user goals

WHY REGULAR?

Adapted from Jeff Gothelf - http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/

“Teams should stretch to get work into that day’s test and use the cadence to drive productivity.”

Jeff Gothelf - http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/

•Recruitment process•Small focused tests•Regular timing or per Sprint•Same day mid-week (not Monday or Friday)

BRING IT ON!

•Select 2 weeks prior to study•“Focus ruthlessly on a small number of the most important problems” – Steve Krug

• Work in Progress• Websites, mobile, products, services• Prototypes: Paper to hi-fi• Concepts, rough ideas, brainstorming• Competing designs (A/B testing)• Comparative studies across market• User research

WHAT TO TEST?

•Make team aware•Invite everyone

• Watch remotely• Recurring meeting invites for stakeholders

USABIL ITY TEST ING DAY

MEASUREMENTS?

•Effectiveness•Efficiency•Learnability•Satisfaction•More…

MEASURE ABIL ITY TO ACHIEVE GOALS

•Success•Time on task (not generally recommended)•% of tasks completed/not completed•Number of steps to accomplish task•Learning•Number of errors•Number of times help consulted•Satisfaction

MEASUREMENTS

MAKE IT REPEATABLE:PLAN AHEAD

•UX • Plan study• Space and equipment• Identify work

PLANNING IS T IGHT

•Test & Observation Rooms•Any location will do:

• Conference rooms• Offices• Quiet corner of cafeteria• Remote

•Time for facilitation, breaks, post-meeting

• No more than 5 per day

PRE-BOOK YOUR ROOMS

http://creativecommons.org/licenses/by-sa/2.0/http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/http://www.flickr.com/photos/raphaelquinet/

•Who are your primary users?• Will they be difficult to recruit?• How expensive is their time?

RECRUITMENT

•Get details:• Education• Computer/Internet use and expertise• Knowledge of topic

•Get them talking• Clear Communicator• Able to express themselves verbally

•People who pass the screener should closely match your user group definition

Guide that helps determine who will participate.

CREATE A SCREENER

•Recruiting: ~$250 per participant (varies)•Compensation: Starts at $50 - $100 and up

• Internally - use swag• “Big fans” – cheaper alternatives ($15 Starbucks card)

Create a Budget

RULES OF THUMB

•Make it easy – buy ahead• Gift cards from retail stores (Target, Starbucks)

•More complex• Amazon Gift Cards• Product credits

•Do not recommend Visa gift cards or similar

HONORARIUMS

•Allows you to focus.

•Good recruiters:• find right participants• give regular updates• take care of directions,

confirmations, incentives, etc.

HIRE A RECRUITER

•Internal resource who REALLY understands who you need•Create a panel (large recruits, less often)

• Internal customer lists, user groups• Schedule as needed• Use cautiously

•Final scheduling ALWAYS by phone• Ask questions that force them to talk• Don’t recruit non-talkers

DO IT YOURSELF

• Pop up on your site:• http://ethnio.com

USE S ITE TO RECRUIT

“We are all only temporarily able-bodied. Accessibility is good for us all.”

•Provide the Spirit of the law (Section 508, WCAG 2.0)

INCLUDE PEOPLE WITH D ISABIL IT IES (PWD)

-@mollydotcom at #stirtrek 2011 via @carologic May 6, 2011

•As many as possible (rarely statistically significant)•Usability Testing Research (in 1990’s)

• 5 from distinct sub-group of the user population will yield 80% of the findings (Nielsen, Virzi, Lewis)

• Assumes expert has reviewed and found obvious issues•Recommend:

• Early tests with 8 – 12 participants per user group• Iterative testing (3 per day, iterate, 3 new participants)

Controversy Abounds

NUMBER OF PARTIC IPANTS

Barnum, Carol M. (Jan. 2003). What’s in a Number? STC Usability SIG Newsletter, Usability Interface. http://www.stcsig.org/usability/newsletter/0301-number.html Retrieved: 20080323

•Your team• Need them to show up• Need to be able to schedule as needed

•Recommend laptop/portable lab• Can be semi-permanent• Ready when you are (minimal setup)

FORMAL LAB APPROPRIATE?

•Screener•Scripts/Guides

• Tasks (same or alternating)• Pre and Post questions

•Consent Forms•Data Collection format

CREATE REUSABLE TEMPLATES

TESTING REMOTELY

Moderated

• No Researcher during study• Minimal effort• High number of participants per

day• Test data compiled usually

within 2 days• Reduced or no ability to ask

questions

REMOTE TEST ING OPT IONS

Un-Moderated

• Researcher is present• Same effort as in-person• Limited number of participants

per day (3-5)• Immediate feedback• Get to the “Why?” by asking

questions

•Lab setup• Robust computer• Screen sharing software

•Participants camera (optional)•Internet on both ends

• Speedy• Separate from phone line

REMOTE - MODERATED

•No “Lab” needed – online software•Participants camera (optional)•Participant’s internet only

•Recommend to complement Moderated Testing

REMOTE – UN-MODERATED

•No travel•Easy to set up•No special room required (advised!)•Can be conducted from almost anywhere•See their computer environment•Drawbacks:

• No F2F• Missed interaction• Technology will fail

BENEFITS OF REMOTE

Un-moderated

• Userlytics• Loop 11• UserZoom• ChalkMark

• GoToMeeting (screen sharing)

• Skype• Morae• Silverback

SOFTWARE

Moderated

DURING THE STUDY

Paper?• Printed Script• Quiet• Analysis may take longer• Great for remote

Computer?• People are less bothered

by typing now• Can you pay attention?• Quicker Analysis

NOTE TAKING

QUICKER – ON B IG PAPER

Issue P1 P2 P3

Search Used Yes No No

Widget 1 Used N/A Used – unsure about

Task 1 Notes 3 – easy 2 – needed help

3 – easy

Task 2 Notes 2 – needed help

2 – easy 2 – needed help


3 – easy Ran out of time


3 – easy Ran out of time

•PO, PM, dev, etc.• Stakeholders must attend to approve changes• Training: Set expectations (no interference)• Take notes

• Make a list of 3 most serious issues

• Help operate software/recording equipment (if in use)

OBSERVERS

•Get them talking•Will get frustrated

• Be prepared - how react• Be supportive • Guide them back to the task• Do not lead

•Listen more than you talk

Relax, they are only human

THEY ’RE HERE…

http://www.flickr.com/photos/chuybregts/ http://creativecommons.org/licenses/by-nc-sa/2.0/

TESTING

•Conduct anytime•One concept tested throughout•Feedback given when the testing is done•Qualitative and quantitative study•Remote and in-person

•“Start earlier than you think makes sense.” – Steve Krug

TRADIT IONAL USABIL ITY TEST ING

•January 16 (Monday)

• Start planning what to test and with whom

• Create Screener

• Start Recruit•January 23

• Determine tasks to test and create guide

• Finish recruiting by Jan. 31•February 2

• 1st Usability Testing Day•February 6th

• Begin planning for Feb. 16th UT Day

2.5 Weeks to Testing

STEVE KRUG STYLE

•3 – 5 users per day• Recruit loosely and grade

on a curve•Stakeholders watch tests – spectator sport•Debrief at end of study•Quick analysis

STEVE KRUG (CONTINUED)

•Small iterative changes• Make it better now• Don’t break something else

•Take something away• Reduce distractions• Don’t add – question it

TWEAK, DON’T REDESIGN

Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. By Steve Krug

•Very quick studies•Very early in the design process (paper or soon after)•Recruiting is basic

• Not familiar with the project• Ideally representative of user group

“Hallway Testing”

GUERILLA STUDY

•RITE is qualitative, exploratory, and conceptual•Creating conversation around elements•RITE is not for:

• measuring performance• validating detailed design decisions

RAPID ITERATIVE TEST ING & EVALUATION

•Most appropriate for:• early requirements gathering• design phase of production

•Initial wireframes used to:• validate information architecture• gather feedback on proposed functionality

RITE OVERVIEW

•Series of small usability tests•Participants attempt tasks on concept•Qualitative user feedback (actions + comments)•New version tested with next day’s participants

RITE OVERVIEW

Test Update Test

1

2

3

High

Medium

Low

Prioritize

Topic RITE Standard Usability Testing

Tested Concept Evolves Mostly static

Suitable stage of project

Must be early Anytime

Dynamics New concepts each day One concept tested throughout

Feedback Different elements throughout

Same elements

Recommendations

Discussed throughout Received in the final report

Data Qualitative Qualitative and quantitative

Stakeholder involvement

Crucial to have daily Preferred

RITE VS . STANDARD USABIL ITY TEST ING

LATER THAT SAME DAY

•After the last session •30 minutes•Room with a whiteboard or paper•All decision makers MUST be present

• If not, wait for UX analysis

DAILY RECAP SESSIONS

•Stay on Topic•Be Constructive•Don’t get distracted by small problems•Intense focus on fixing most serious problems first

GUIDEL INES

•Identify top 5 or 10 most serious issues• Top 3 from each list• Prioritize from lists• Make assignments for next piece of work• Stop

GOAL

Adapted from: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. By Steve Krug

•Quick analysis to quick decisions•Discuss:

• trends seen• concerns• recommendations• prioritize changes for the next round• list lower priority changes for future iterations

DEBRIEF WITH TEAM

MAKE USEFUL & USABLE RECOMMENDATIONS -

QUICKLY

•Analysis up to ~1.5 x the facilitation time•Identify repetition and patterns•When found, continuation:

• Adds cost • Delays reporting• Low probability of many new findings

Read “between the lines”

ANALYSIS

•Know what you’ve got • Sort, reorganize, review, repeat• What refutes your expectations?• Surprises?• Outliers?

TRANSFORM DATA

•Reduce, reuse•Use email, IM, etc.•Put it on the wall

• Must be easy to understand• Quickly absorbable

DOCUMENT WITH LESS TREES

•Your Communication Goal:• Think about audience• How will it be used?

•Include:• Study’s Goal• Tasks attempted• Who observed• What planned to fix and assignment• Future research/enhancements

EMAIL OR ONE PAGER

SHARE WHAT YOU LEARN

Information radiators• Test findings• Artifacts• Personas

MAKE USERS V IS IBLE

TRANSFORM DATA

They liked this!

Make this go away!

•Create useful findings and recommendations• Show screenshots of issues• Where possible show solutions

•Think about audience• How will it be used?

•Plan out your goals •Recruit participants•Remote or in person?•Look for patterns•Easy-to-use findings and recommendations

REVIEW

DO USABILITY TESTING EARLY & OFTEN

easy & effective usability testing at codemash 2012

Self Improvement

s n t i

t o t e s t

i t y t e s t i n g

pa r t i c i pa n t

mistakes http

s c r e e n e r guide

satisfied users http

jeff gothelf http