lecture #3.3 usability tests *39tur summer 2011/2012

Lecture #3.3USABILITY TESTS*39TUR Summer 2011/2012

(2)

Who is the “customer” of the usability tests?

Who asks you to do the usability test?– Designers– Software engineers– Management– Government agencies

(3)

How well do they specify what to do?

Thorough assignment– “Here’s a detailed overview of what situations to test. We want to

test such and such users, in such and such conditions.”

Partial assignment– Specific user group: “We would like to know how well this works

with {elderly | children | violists }.”

… Vague assignment

– “Find out whether it works.”– In such case it is the responsibility of the tester to refine the

assignment.

Terminologická poznámka

“Tester”– Rozpor ve významu– Ten, s kým se testuje (dobryweb.cz, h1.cz)– Ten, kdo organizuje a provádí test (správná)

Lepší termíny– “Participant”– “Účastník testu”

METHODS OF USABILITY TESTING

(6)

Usability Testing

Usability testing– Main goal: Looking for design problems

Acceptance testing– Refers to user satisfaction. A subset of the usability

testing.• “Will users like it?”

– Make sure the system meets expected criteria from the user side

• “80% customers need 1—3 minutes to withdraw amount of $50.”

(7)

Methods

Non-user based (without users)– Expert review– Heuristic evaluations– Cognitive walkthrough

User-based (with users)– User surveys– Ethnographic observations– Usability engineering

(8)

Methods with Users

Observing users in their own environment– Their office, etc.– Natural and therefore realistic conditions

• Environment plays very important role (ringing phones, traffic on the streets, …)

• The user is accustomed to these conditions

– Difficult to organize, time consuming– Results can not be generalized

(9)

Methods with Users

Controlled experiments– Controlled (laboratory) environment– Controlled conditions

• Limited influence of external stimuli

• No unwanted distractions by other people

• No unwanted distractions by software running on background

– Repeatable• The same procedures will give the same results

– Non-realistic conditions• problem with ecological validity

(10)

User Surveys

Questionnaires with questions oriented towards expected usability problems

Quantitative– Get precise measurement of one particular aspect

Qualitative– Open-ended questions

– Check how much you agree with the following statements:– “It is easy to enter the telephone number”– [strongly agree] [agree] [neither] [disagree] [strongly diagree] – “It is easy to setup the owner’s address”– [strongly agree] [agree] [neither] [disagree] [strongly diagree]– “The menu item ‘Games’ is easy to find”– [strongly agree] [agree] [neither] [disagree] [strongly diagree]

(11)

User Surveys – Quantitative

Check how much you agree with the following statements:“It is easy to enter the telephone number”

[strongly agree] [agree] [neither] [disagree] [strongly diagree] “It is easy to setup the owner’s address”

[strongly agree] [agree] [neither] [disagree] [strongly diagree]“The menu item ‘Games’ is easy to find”

[strongly agree] [agree] [neither] [disagree] [strongly diagree]

Your age: ………

Your income: ………+ Easy to process+/– Limited amount of possible answers

(12)

User Surveys – Qualitative

What are the biggest problems you find whileusing the telephone XX-1234?

………………………………………………………………………………………………………………

What functions do you use most frequently?………………………………………………………………………………………………………………

– More difficult to process+/– Intended to map topics that were not previously studied

(13)

User Surveys

Pros– Inexpensive– Can be done remotely

• “While waiting for the video to load, why don’t you fill out our little questionnaire?”

– Can provide data on the current trends Cons

– Relies only on the data provided by the users– Recruiting the right people is a problem

• Self-selection bias• Add some screening questions• Provide incentives

(14)

Ethnographic Observation

Observing users in their environment, without any intervention– Can be rural India, can be office of the US government.– E.g.: “Find out how cell phones are used in rural areas

of India.”– E.g.: “Find out how / whether the elderly people spend

their day. Could they use instant messaging of some sort?” (ICQ, Skype…)

The observer stays with the users and watches them how they use (struggle using) some system.

(15)

Ethnographic Observation

Pros:– Based on the reality

Cons:– Needs a thorough preparation– Not an interactive method– Possible ethical problems– Expensive

(16)

Usability Engineering

Nielsen (1983) Assessing the usability of a design

Observing the users using the system in a simulated environment– Aspects of the real world are simulated– Users work on predefined, but realistic tasks– Users are observed (measured) while carrying out the

tasks– Identification of the problematic parts

(17)


Are the results of such a test comparable to the results obtained from the “real-world observation”?– Selected users are being tested– The tasks are defined by the experimenters– Users are not in their natural “habitat”– Different social settings

Solutions– Use the real users– Make the environment as real as possible– Consult the tasks with the users

(18)


Make the tasks realistic Make the tasks ecologically valid

LAYOUT OF A USABILITY LAB

(20)

Usability Lab Usability Lab – – usability testingusability testingObserver roomObserver room

(UI designers(UI designers, , programmersprogrammers, , test organizers)test organizers)Participants roomParticipants room(test participant(test participant, , moderator)moderator)

(21)



(22)



(23)



6-126-12

(24)



TasksTask 1: xx xxx xx x xxx xx xxx.

Tasks

Task 1:In few minutes explore the new IDE environment. Please think aloud during the exploration process.

Task 2:Pleas create and run simple mobile application "Hello World", which will display "Hello World!" message on the display of the cell phone emulator.

Task 3:Your colleague finished the application GoodBye. Your complete this application wit

(25)




Tasks

Task 1:In few minutes explore the new IDE environment. Pleas think aloud during the exploration process.

Task 2:Please create and run simple mobile application "Hello World", which will display "Hello World!" message on the display of the cell phone emulator.

Task 3:Your colleague finished the application GoodBye. Your complete this application wit

(26)



I am trying to create new

project


Lo

gT

ask 1: xx xxx xx x xxx xx xxx.

Test Session Log

Task 1:12:01 start12:07 end

Task 2:12:09 start12:10 creates new project

(27)




project


Záp

isÚ

loh

a 1: xx xxx xx x xxx xx xxx.

I can't find appropriate menu item

Test Session Log


Task 2:12:09 start12:10 creates new project

12:12 can not find appropriate menu item

(28)




project


Záp

isÚ

loh


I expected item "New Project" in menu

"File".

I can't find appropriate menu item

Test Session Log


Task 2:12:09 start12:10 creates new project12:12 can not find appropriatemenu item

12:13 expects "New Project" in menu "File"

(29)




Záp

isÚ

loh


Test Session Log


Task 2:12:09 start12:10 creates new project12:12 can not find appropriatemenu item12:13 expects "New Project" inmenu "File"12:27 end

Task 3:12:31 start

(30)



Záp

isÚ

loh


Rep

ort

1. Go

al xx xxx xx x xxx xx xxx.

Final report

1. Test goal2. Summary3. Results

3.1 Issue 1Problem descriptionSolution proposal

3.2 Issue 24. Conclusion

(31)

We do need a lab!

Everything is at hand– The software that is tested, the tools for recording– The experts are present– The stakeholders can witness the process

A chance to observe and record non-obvious findings– Survey:

• User fills out: “I can use the function X to do Y.”

– Usability engineering:• User says: “I can use this function X to do Y.”• Experimenter: “Show me.”• User: “Sure, I just …” (struggles) “uh … actually, I can not!”

lecture #3.3 usability tests *39tur summer 2011/2012

Documents