boston april 18, 2013 rocket surgery made easy: the do-it-yourself usability testing workshop
TRANSCRIPT
BostonApril 18, 2013
Rocket Surgery Made Easy:
The Do-It-Yourself Usability Testing Workshop
© 2001 Steve Krug
Who is this guy, anyway?
Steve Krug (steev kroog) (noun) 1. Son, husband, father 2. Resident of Brookline3. Usability consultant
Advanced Common Sense Me and a few well-placed mirrors Corporate motto: “It’s not rocket surgery™”
Nice clients Lexus.com Bloomberg.com Technology Review
© 2001 Steve Krug
I get to work at home
© 2001 Steve Krug
I get to work at home
© 2001 Steve Krug
I get to work at home
© 2001 Steve Krug
My intention for today
Give you some practice so you’re comfortable testing
Try to answer all your questions so you have no reason left not to test
© 2001 Steve Krug
This morning
Why do usability testing? Steve does a demo test Six Maxims Writing tasks and scenarios
© 2001 Steve Krug
This afternoon
You do your first practice tests You do another practice test (maybe) Assorted topics Lingering questions Giveaways and feedback
© 2001 Steve Krug
Ground rules
Tell the driver to speak up, if necessary Interrupt ANYTIME with questions I’ll answer questions about anything
except that brief period during the late 70’s
© 2001 Steve Krug
Anybody here from out of town?
Graphic designers Information architects Developers/programmers “Marketing” Usability ______ Project managers Writers/editors Check signers Other? Left-handed?
© 2001 Steve Krug
Anybody here from out of town?
Your experience with usability testing Have conducted tests (facilitator)? Have observed tests? Have read usability test reports?
Your organization’s use of testing Never? Right before (or right after) product ships? Routine (several times during development)? Farmed out? Have your own lab (and white coats)?
© 2001 Steve Krug
One more question
What are the biggest obstacles to your doing testing?
Who’s read the book(s)?
DMMT? Rocket Surgery? Watched the video? Don’t worry, be happy, ask questions
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
So…
Why usability testing? Ten years ago, I realized something…
© 2001 Steve Krug
© 2001 Steve Krug© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
“My ideal home page,” as told by…
© 2001 Steve Krug
“My ideal home page,” as told by…
And now, a live demonstration
© 2001 Steve Krug
© 2001 Steve Krug
A brave volunteer?
We’ll try an actual test It’s painless It’s brief You’ll get a round of applause when we’re
done
Qualifying criteria: Have used a Web browser English-speaking adult Doesn’t work at Fresh Tilled Soil
During the test You are observers Jot down top 1 or 2 problems you observed
Debriefing
What were the most serious problems? Observed problems
© 2001 Steve Krug
© 2001 Steve Krug
DIY usability testing (nutshell version)
Three users You’ll find more than you can fix
No lab or mirrors Set up a monitor in another room so the
development team can watch
Record with Camtasia or Morae (Techsmith) or various Mac
products
No stats, no exit questions, no faux validity
No big honkin’ report Debrief over lunch
The maxims
Six of them I could (and have) talk about them all day Questions highly encouraged
What seems like it might not work for you?
© 2001 Steve Krug
© 2001 Steve Krug
A morning a month, that’s all we ask.
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
When this happens: Say this:You’re not absolutely sure you know what the user is thinking (see below).
“What are you thinking?”
“What are you looking at?” (for variety)
“What are you doing now?” (e.g., if you think they’re being silent because they’re reading)
Something happens that seems to surprise them. For instance, they click on a link and go “Oh” when the new page appears.
“Is that what you expected to happen?”
They’re trying to get you to give them a clue. (“Should I use the ___?”)
“What would you do if you were at home?”
“What would you do if I wasn't here?”
The participant makes a comment, and you’re not sure what triggered it.
“Was there something in particular that made you think that?”
The participant suggests concern that he’s not giving you what you need.
“No, this is very helpful.”
“This is exactly what we need.”The participant asks you to explain how something is supposed to work. (“Do these support requests get answered right away?”)
“I can’t answer that right now, because we need to know what you would do when you don’t have somebody around to answer questions for you. But if you still want to know when we’re done, I’ll be glad to answer it then.”
The participant seems to have wandered away from the task.
“What are you trying to do now?”
© 2001 Steve Krug
Start earlier than you think makes sense.
Incorrect thinking
© 2001 Steve Krug
Correct thinking
© 2001 Steve Krug
© 2001 Steve Krug
Recruit loosely and grade on a curve.
© 2001 Steve Krug
Naturally, we need to test people who are just like our target
audience. … people who are a lot like
our users.
… people who actually use our
site.
Representative users!
Real users!
© 2001 Steve Krug
© 2001 Steve Krug
Make it a spectator sport.
© 2001 Steve Krug
© 2001 Steve Krug
Focus ruthlessly on a small number of the most important problems.
What’s funny about this?
Show of hands: Have you ever gone to a Web site and run into a serious usability problem?
Did you find yourself thinking “How can they not have noticed this? And fixed it?”
Did you go back months later and it was still there?
© 2001 Steve Krug
The problem is, testing works
If you’ve done any testing, you know it works
Uncovers lots of problems quickly But I’ve finally realized this is part of the
problem You can find more problems in a day than
you can fix in a month
© 2001 Steve Krug
© 2001 Steve Krug
Problems you can find with just a few test participants
Problems you have the resources to fix
Things I have learned
It’s easy to get seduced into fixing the easier problems first
As a result, the most serious usability problems often remain for a long time
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
When fixing problems, always do the least you can do™.
© 2001 Steve Krug
Your motto
When fixing usability problems, your motto should be: What’s the smallest change we can make
that we think might solve the observed problem?
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
Nothing New Under the Sun Department
© 2001 Steve Krug
Choosing tasks to test
What do you have to show? Try not to let this limit your thinking, though
You can get a long way with A sketch or a few “comps” Linked wireframes to test navigation HTML of a few pages that let you complete a
task as long as you don’t stray See Carolyn Snyder’s Paper Prototyping
© 2001 Steve Krug
Choosing tasks to test
What do you wake up thinking about in the middle of the night?
What tasks are people likely to do? What tasks are crucial?
…to the user and your business model
Whenever possible, keep it real Free-range browsing tasks are a good thing Bad: “Buy a gift cupholder for under $35.” Better: “Order a book you’d like to have”
Tasks vs. Scenarios
Task: “Apply for a doctoral program at HBS”
Scenario: “You’ve got an MBA, and after a lot of research
you’ve decided to enter the doctoral program at HBS in Science, Technology & Management.
Apply for admission to the program.”
A scenario... Provides some context (“You are...”, “You need
to...”) Just enough; DON’T get carried away Supplies specific information the user would
actually have © 2001 Steve Krug
© 2001 Steve Krug
Writing the scenarios
Don’t telegraph it Avoid using words that will appear on-screen Bad: “Customize your LAUNCHcast station.” Better: “Choose the kind of music you want to
listen to.” Can be the hardest part
Make it unambiguous Misunderstandings waste time and don’t
[usually] produce useful insights
Keep it short Trim any detail that doesn’t contribute
Piloting the test will help
Exercise: Preparing tasks
Jot down 3-7 of the most important tasks people need to do on your site (4 min.) E.g., “Find directions to your nearest Bank of
America branch”, “Apply for a doctoral program”
Choose one to work on Write a scenario for this task (10 min.)
A short paragraph “Don’t use Search” is OK Write it out so you can read it verbatim
© 2001 Steve Krug
Now send me your scenario(s)
Go to
bit.ly/BOSTONSK
Click the orange “Post a new message” button
Use your name to sign in (so we can identify them)
Type the URL you’re testing Type your scenario
© 2001 Steve Krug
Exercise: Practice test
Pair up with someone who has a laptop Take turns as facilitator/participant 15 minutes each
© 2001 Steve Krug
Exercise: Practice test
Read the introduction from script (2 min.) Sign recording consent form Start Camtasia (or equivalent) if you have it
Ask them the five questions (2 min.) Have them look at Home page (<2 min.) Have them do your task (8 min.)
Give them the task Keep them thinking aloud
Ask probing questions (2 min.) Thank them (30 sec.)
Save your Camtasia file© 2001 Steve Krug
How’d it go?
© 2001 Steve Krug
The debriefing
Over lunch (or dinner, or breakfast) Right after the three test sessions Objective: Deciding what you’re going to
commit to fixing before the next round of testing
© 2001 Steve Krug
The debriefing
Go around the room Everyone contributes from their list of nine
problems Write on easel pad Leave some space for
improvements/amendments People can say “Me too!” Treat all contributions with respect Not discussing yet Stick to observed problems!
© 2001 Steve Krug
The debriefing
Decide which are most serious Some magic happens here Voting/Dictatorship Not usually as hard as it seems BECAUSE
THEY ALL SAW THE SAME BEHAVIOR Number them
Copy the numbered list Ten is probably enough Leave space in between
© 2001 Steve Krug
The debriefing
Start at the top Work down the list Come up with rough idea of how you’ll fix
them who will do it the resources required
When you’ve allocated the resources you can commit in next month, STOP! Tear off the rest of the list Crumple it up Throw it away
Thanks to Susan Weinschenck © 2001 Steve Krug
Mobile testing
Testing things on mobile devices Apps and mobile sites
Head for Slideshare Belen Barros Pena & Bernard Tyers http://www.slideshare.net/beleniq/do-it-yours
elf-mobile-usabilitytesting-at-euroia-2010 “The McGyver’s of Mobile Usability Testing”
© 2001 Steve Krug
What the difference?
Basically the same Except for a ton of issues
How does the facilitator see the screen? How do the observers see the screen? Fingers/No fingers? (Camera view or screen
mirroring) If camera, attached to device or strap the
user to a chair? Screen recording? What devices? User’s own or supplied by
you? Test in context? If so, are there observers?
© 2001 Steve Krug
My take on it
Two useful tools Airplay (mirroring iOS devices to PC or Mac
via WiFi) Screen recorders for iOS
UX Recorder and Magitest appsMany limitations (browser only,
performance issues), but valuable if “walking around” testing is needed
See my recent blog postwww.someslightlyirregular.com
© 2001 Steve Krug
My take on it
Observers are important Fingers are important for observers
Faces, not so much
My ideal finger-view camera Featherweight, unobtrusive, stable, reliable Manual focus Inexpensive (<$50)
© 2001 Steve Krug
Lightweight webcam
© 2001 Steve Krug
+ Lightweight Clamp and Gooseneck
© 2001 Steve Krug
= Brundlefly
© 2001 Steve Krug
My take on it
Optimal setup Webcam > facilitator’s laptop via USB Facilitator watches webcam on laptop (using
AmCap or similar program) Laptop mirrored to observers via screen
sharing (GoToMeeting, Webex, etc.) Recording is made on observation room
laptop (Camtasia, Morae, etc.)
© 2001 Steve Krug
© 2001 Steve Krug
A great tip
© 2001 Steve Krug
And the companion volume…
© 2001 Steve Krug
© 2001 Steve Krug
© 2013 Steve Krug