usability testing: targeting mobile devices

Introduction to User Testing Wayne Pau, Emerging Technologies SAPMar28, 2013 Internal

Wayne Pau, Emerging Technologies SAPFeb 4, 2013 – MEIC’s Mobile Accelerator Programme – Workshop #6

Introduction to User TestingImplementing Quick & Effective User Testing

© 2012 SAP AG. All rights reserved. 3Internal

My Apology Slide

“If I'm curt with you it's because time is a factor. I think fast, I talk fast and I need you guys to act fast if you wanna get out of this. So, pretty please... with sugar on top…”

- Winston Wolfe (Pulp Fiction)

Background InfoGiving credit to Steve Krug


Buy this book…

• Steve Krug

• 2nd Book

• Cheap & Dirty Usability Testing

• 161 pages


… then buy this next… (and read it first)

• Steve Krug

• 1st Book

• Web-Focused

• 216 pages


Even more info…

• Website:

Advanced Common Sense

http://www.sensible.com/




Steve Krug – Recorded Sample

http://www.rocketsurgerymadeeasy.com/

http://www.youtube.com/watch?feature=player_embedded&v=QckIzHC99Xc

All of Chapter 2 in Rocket Surgery Made Easy








The “Basics”Some general info about Usability Testing


What User Testing is Not…

• Not statistically relevant Quantitative Research

• Not a Market Focus Group

• Not for “Baking-Off” UI Choices

• Not mining for new UI Designs/Patterns

• Not Contextual Design or “Voice of God” interviews


Why do User/Usability Testing

1. No Product is perfect. Everything can be Improved.

2. Serious problems are easy to Spot. Encountered by most Users.

3. User-Centered approach makes for better products.


How to do User Testing

• One Morning per Month

• Find 3x “Participants for 1 Hr session

• Find 3x Tasks, expand into Scenarios

• Record/Broadcast

• Debrief in Afternoon

Recruiting Users


# of Subjects

• 3 or 4 for 2+ rounds

• 3 or 4 allows testing

• AM, debrief in PM

• 2x Rounds w/ 3 > 1x round with 6

• Recruit loosely, grade on a curve


Recruitment - Exceptions

• If you site is only used a particular population/relevence

• If you subject has conflicting or divergent interests

• If you app is esoteric and needs domain expertise


Recruitment – Good Ides

• Offer good Incentive (Gift Cert)

• Keep invitation simple

• Don’t offer too much info

• Ask Relatives/Friends/Family

• Everyone was Newbie once

Testing ScriptA template for Usability Session


Script Breakdown

1. Welcome

2. Questions (Warm-up)

3. Acclimatizing (Familiarizing Participant)

4. Tasks (Prepared Scenario)

5. Probing (Digging into issues)

6. Wrapping Up (Close & Thanks)

Welcome


Welcome – Part 1

Hi, . My name is Wayne Pau, and I'm going to be walking you through this session.

You probably already know, but let me explain why we've asked you to come here today: We're testing a web site that we're working on to see what it's like for actual people to use it.

I want to make it clear right away that we're testing the site, not you. You can't do anything wrong here. In fact, this is probably the one place today where you don't have to worry about making mistakes.


Welcome – Part 2

We want to hear exactly what you think, so please don't worry that you're going to hurt our feelings. We want to improve it, so we need to know honestly what you think.

As we go along, I'm going to ask you to think out loud, to tell me what's going through your mind. This will help us.

If you have questions, just ask. I may not be able to answer them right away, since we're interested in how people do when they don't have someone sitting next to them, but I will try to answer any questions you still have when we're done.

We have a lot to do, and I'm going to try to keep us moving, but we'll try to make sure that it's fun, too.


Welcome – Part 3

You may have noticed the camera. With your permission, we're going to videotape the computer screen and what you have to say. The video will be used only to help us figure out how to improve the site, and it won't be seen by anyone except the people working on the project. It also helps me, because I don't have to take as many notes. There are also some people watching the video in another room.

If you would, I'm going to ask you to sign something for us. It simply says that we have your permission to tape you, but that it will only be seen by the people working on the project. It also says that you won't talk to anybody about what we're showing you today, since it hasn't been made public yet.

Do you have any questions before we begin?


Welcome - Tips

• Stick to the script, stick to the script, stick to the script…

• Try to remain neutral & calm

• Test Participant (Not Subject. They are not being tested)

Acclimatizing


Questions – Part 1

Before we look at the site, I'd like to ask you just a few quick questions. First, what's your occupation?

Good. Now, roughly how many hours a week would you say you spend using the Internet, including email?

How do you spend that time? In a typical day, for instance, tell me what you do, at work and at home.

Do you have any favorite Web sites?


Questions – Part 2

Now, finally, have you bought anything on the Internet? How do you feel about buying things on the Internet?

And what have you bought?

OK, great. We're done with the questions, and we can start looking at things.


Questions - Tips

• Build some rapport with your Participant

• Assist in your “Grade on a Curve”

• Get Participant Talking…

Acclimatizing


Acclimatizing

OK, great. We’re done with the questions, and can start looking at things.

First I’m going to ask you to look at this page and tell me what you make of it: what strikes you about it, whose site you think it is, what you can do here, and what you can do here, and what it’s for. Just look around and do a little narrative.

You can scroll if you want to, but don’t click on anything yet.


Acclimatizing - Tips

• Get them used to listening to instructions

• Get them used to verbalizing

• Get them use to “thinking out loud”


Tasks

Now I’m going to ask you to try doing some specific tasks. I’m going to read each one out loud and give you a printed copy.

I’m going to ask you to do these task without using Search. We’ll learn a lot more about how well the site works that way.

And again, as much as possible, it will help us if you can try to think out loud as you go along.


Tasks – FoodNetwork.ca


Tasks #1 – FoodNetwork.ca

Assume you are big fan of Anna Olsen and love all her shows. You

are going to be home tomorrow all day and would like to determine

when she is on first on FoodNetwork tomorrow. If possible, find out:

• What TV series it is (she has multiple)

• What is the Episode About

• What recipes will be shown (in case you want follow along)



Assume you are big fan of Anna Olsen and love all her shows. You are going to

be home tomorrow all day and would like to determine when she is on first on

FoodNetwork tomorrow. If possible, find out:

• What TV series it is (she has multiple)

• What is the Episode About

• What recipes will be shown (in case you want follow along)



You’ve recently come back from a short vacation to New Orleans. You had a

great time and especially loved the Cajun food. Check FoodNetwork.ca for

possible main course dinner dishes that you can try at home. Assume you will

looking at following criteria to help you narrow it down:

• How it was rated

• You like Fish over Red Meat

• You really Blacken Anything



You’ve had a rough week and are looking at cheering yourself up

with some food. However you still want to eat healthy and really

don’t like nuts (actually you could be allergic). Can you find a good,

quick recipe that would entice you?


Tasks - Tips

• When in doubt *SAY NOTHING*

• Use every permutation of “uh huh” or “OK” possible

• Aim for “somewhat” pleased demeanor

• Deflect ( “What would you do if I was not here?”…)

• Mine leading non-verbal clues (“Is that what you expected?”)

Probing


Probing

Thanks, that was very helpful.

If you’ll excuse me for a minute, I’m just about to see if the people on the team have any follow-up questions they’d like me to ask you.


Tasks - Tips

• Relax “hands-off” approach

• Ask “Why”?

• Avoid asking for “solutions” or “redesigning on the spot”

Wrap-up


Wrap-up & Thank-you

Do you have any questions for me, now that we’re done?

Thanks. That was exactly what we need, it’s been very helpful.

User Test Results


Real Testing World

• Review results right away

• Triage the information

• Find a Fix (or approach)

• Do the *LEAST* you can do

to fix the problem


7 Reasons to Tweak (vs. Rebuild)

1. Cheap & Cost effective

2. Less work & resource intensive

3. Less Stressful

4. Implemented Quicker

5. Small & more likely to be scheduled

6. Less Risky than Overhaul

7. Might know the issue, but not the “perfect” solution


Wayne’s Issue Triage

1. Is it MVP? Can we “Dump” the culprit?

2. Is there a Small Tweak we can do?

3. Was it effective enough? (Do we need to Tweak the Tweak?)

4. Did we Breaking anything more Important?

5. Is it a new feature? DON’T ADD ANYTHING!!!


Something is Better than Nothing…

“Perfect is the Enemy of Good”

– Voltaire

“A good plan executed today is better than a perfect plan executed tomorrow.”

- General George Patton

POGE – Principle of Good Enough

Thank you

Contact information:

Wayne Pau ([email protected])Emerging Technologies

mailto:[email protected]

usability testing: targeting mobile devices

Design

sap ag

testing script

web site

actual people

usercentered approach

x tasks

usability session

script breakdown1