© idc, iit bombay 1. 2 baddesigns.com © idc, iit bombay 3

BadDesigns.com

World Is Full of Unusable Objects

Home assignment– Find 3 unique usability problems in the day-to-day objects

that you have observed or encountered– Describe them as 3 slides in a presentation

Usability Evaluation

Objective

To identify usability problems in an interface– To improve the design – formative evaluation– To compare competitive products or as acceptance test

– summative evaluation

Many Methods

Heuristic evaluation User Testing

Thinking Aloud Protocol Card sort Feedback from real life usage

– Observational field studies– Logging actual use– User Feedback– Questionnaires and Interviews– Focus groups

Usability Evaluation Methods

Heuristic evaluation– Jakob Nielsen, Usability Engineering, Morgan Kaufman,

For use by a shopkeeper in a kirana shop to make a bill– Find as many usability problems

as you can in 5 minutes in the Standard Windows calculator

For each problem – State it succinctly (if you

thought of a design idea, tell us what problem do you solve)

– Identify a (potential) user experience goal that was not met

– Rate it for severity– Identify layer of user

experience

Analyse the alarm app shown in this video for use by a patient educated till class VII as reminder tool for his medication – Find as many usability problems as you can in 5

minutes

Analyse the app shown in this videohttp://www.youtube.com/watch?v=s94JxnVwJtI

Analyse the alarm app shown in this video for use by a patient educated till class VII as reminder tool for his medication – Find as many usability problems as you can in 5

minutes For each problem

– State it succinctly (if you thought of a design idea, tell us what problem do you solve)

– Identify a (potential) user experience goal that was not met

– Rate it for severity– Identify layer of user experience where the problem

occurs

Heuristic Evaluation Technique

5-10 evaluators examine the product and identify usability problems

It helps to have a mix of usability experts, domain experts and users– Single / double experts– Same evaluators provide consistency across versions

and save time– New evaluators provide fresh eyes

Heuristic Evaluation Technique

Give assistance regarding the domain knowledge– Brief about users, personas, scenarios, primary goals,

and usability goals– Evaluators may be asked to focus on parts of the

product / specific personas Give a checklist of

– HCI heuristics and principles– Platform standards, common practices– Domain standards, common practices

Heuristic Evaluation Technique …

Each evaluator has the freedom to choose his/her method of evaluation– Each evaluator evaluates the product independently– Evaluators should go through product 2-3 times

First iteration– Look for problems novice might face– Analyze Surface and Skeleton issues

Second iteration– Use primary goals, usability goals, heuristics, domain

standards, platform standards to identify problems– Look for conceptual problems, analyze Structure

Third iteration– Look for what is being missed out – Question Scope and Strategy issues

Sometimes, an observer notes the findings– Each of us will note for ourselves here

Observer may give hints about product use – These are evaluators, not users– Save time, capture the problem, and move on– Help only when evaluators ask, after noting problems

After the evaluations, have a debriefing meeting to compile findings

The debriefing meeting also produces design suggestions– Make them sound like suggestions rather than decisions– Do not express only a design idea, point to the problem

first Typical debriefing sessions last 1-4 hours

– Produce preliminary report during the debriefing meeting

– Fine-tune later

Heuristic Evaluation Output

Output of heuristic evaluation is a list of usability problems

Report formats– Introduction, product evaluated, method used– Scenario based problems

~ Enter the cubicle, swipe the card, enter PIN, check balance, withdraw cash, collect cash, card and receipt, exit the cubicle

– Screen based problems in callouts~ Idle screen, test keyboard screen, login screen,

password screen, main menu Common Industry Format (CIF)

~ ISO/IEC 25062:2006

Heuristic Evaluation Some Numbers

5 evaluators find about 75% of the problems 10 evaluators find about 90% of the problems

– HE is prone to reporting false alarms– Don’t negotiate for too long unless it is a very important

issue– If someone thinks it is a problem, capture it

User Testing

A representative set of tasks Representative users try to perform these tasks An experimenter conducts the experiment

– Observes users– Captures, interprets and prioritizes problems

Planning a User Test

Determine the key question that the UT should answer~ Should we design an IVR? Can low-tech savvy users

use IVRs? ~ Is the design OK? Should we start the development?~ Can all voters use the new EVM? Can we use it in the

next election?– Subset of usability goals and persona goals

Determine the key question that the UT should answer Determine test tasks

– Determine training to be given prior to tasks– Provide a scenario description– Give users a goal to achieve, not instructions to execute– Avoid leading instructions / hints / questions– Set ‘error traps’ and planned disturbances– Try to provide early success experiences

Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria

– Determine the output of the UT– Identify performance measures

~ Quantitative~ Success, time, errors, efficiency, satisfaction~ Qualitative~ Ability to handle new situations~ Conceptual understandability of the product~ Questions asked by users~ Training procedure

>> Output of a user test

User Testing Output

Qualitative output~ What is the first reaction?~ What is confusing? What do users ask? Why?~ What do the users like / hate most?~ What features are users likely to use?~ What are user priorities?

– Good for formative evaluation

User Testing Output

Qualitative output Task success

~ Find the share price of Reliance~ Find a good insurance policy

– Levels of success ~ Yes / no~ Successful in first attempt / second attempt / with

User Testing Output

Qualitative output Task success Task time

– Tell the participant to be as fast as possible – But avoid declaring

~ “You took 6.3 seconds for that task. Great show.”– Decide when to stop the timer– To probe or not to probe?– Include unsuccessful tasks in timing?

User Testing Output

Qualitative output Task success Task time Errors

– Error “an action of the user that does not accomplish the desired goal”

– Severity of errors “the more difficult it is to recover from the error, more sever it is”

Few users face the problem / problem occurs infrequently

Many users face the problem / problem occurs frequently

Easy to recover Low Medium

Difficult to recover Medium High

User Testing Output

Qualitative output Task success Task time Errors Efficiency

– Efficiency = success + time– Number of mouse clicks– “Lostness” – Smith

User Testing Output

Qualitative output Task success Task time Errors Efficiency Satisfaction scores

– What to measure?~ Expectation, ease, satisfaction, time (no watch),

smiles / frowns – How to measure?

~ Likert scales, semantic differential scales– When to measure?

~ Pre-test, post-test, in the field– Try to make it look anonymous – online reduces bias

Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria

– This will be the output of the UT~ Qualitative~ Quantitative: success, time errors, efficiency,

satisfaction– Correctness criteria for each task– Things to measure– Things to watch out for– When to help users

Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria Invite / recruit appropriate users

– Matching personas, target markets– Covering independent variables– Professional recruiters– Screener– Telephonic screening

User Testing Screener

Make it short – Less than 20 questions that take about 5 minutes to run

through. Be clear and specific

– Avoid questions that can lead to ambiguous answers Avoid unnecessary questions

– A screener is not an interview– Each question should eliminate some users

Order questions from general to specific – Try to eliminate more people sooner

Communicate the format of the UT and the expected time commitments – Clearly communicate if the user needs to be prepared

in some way Plan on drop outs and no-shows

– Recruit 20% more users than you need

Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria Invite / recruit appropriate users Prepare the infrastructure

– Carry out all test tasks to ensure readiness– Carry out pilot tests for practice

User Testing Process

Introduce the user to the topic and put him/her at ease– Explicitly say that it is the product that is being tested– Explain that the product is new and untested– Promise confidentiality, inform about any recordings– Avoid large crowds, conspicuous cameras, interruptions other

than the planned ones Give test tasks one at time Give hints to only if the user is stuck for a long time

– Or if they run into a known usability problem After the tests, debrief the user clarify any doubts Review test plan before repeating with another user

Interview Structure

Start– Break ice – factual questions– Ask 1-2 open ended questions

Middle– Give tasks, one at a time– Contextualize findings – concrete, detailed data– Keep track of test goals

End– Debrief, answer questions, discuss insights– Ask for suggestions

Let the user talk – Avoid interruptions– Try to build a partnership

Questions for Non-directed Interviewing

Don’t imply the answer~ “We put in a lot of effort to design this product. Do

you like it?”

Don’t imply the answer Questions should be non-judgmental

~ “Don’t you think that calorie advice will be more useful as a graph?”

~ “What do you learn from this graph?” or “With this tool select an appropriate dessert for the family.” then observe and probe as appropriate

Don’t imply the answer Questions should be non-judgmental Keep questions “open-ended”

~ “Which of these features is most important to you?” implies “At least some features should be important to you.”

~ “Rate the importance of these features on a scale 1-5”

Don’t imply the answer Questions should be non-judgmental Keep questions open-ended Focus on experience, not extrapolation

~ “Do you think car pooling is a useful feature?” Implies “In the universe of all things, do you think that someone somewhere will find carpooling useful?”

~ “Given your lifestyle and the amount of commuting you do, do you think carpooling valuable to you right now?”

Don’t imply the answer Questions should be non-judgmental Keep questions open-ended Focus on experience, not extrapolation Further, focus on immediate experience

~ “Is a in-car GPS interesting to you?” Implies “Will a in-car GPS facility be of interest to you at some time in your life?”

~ “If your car is equipped with this GPS facility, would you use it?” Then, if yes “How will you use it?”

Don’t imply the answer Questions should be non-judgmental Keep questions open-ended Focus on experience, not extrapolation Further, focus on immediate experience Focus a question on a single topic

~ “How will you use the MP3 option at a party and while driving?”

Running a Non-directed Interview

Define ambiguous terms Restate answers to verify (= share interpretation)

– Listen to user’s questions carefully Use artifacts to ground discussions Don’t force opinions

– Never say that the user is wrong – you are here to find out what the user thinks, not to convince him / her about your opinions

Follow up with examples to probe– But wait for an undirected answer first

Review interview tapes– You will find your own ways of improvement

What can go wrong?

Reliability– “Can we get the result if we repeat the test”– 10x difference in individual computer usage speeds– Background of users

~ Access, age, familiarity with UTs

What can go wrong?

Reliability Validity

– “Does the test reflect the usability issues of interest”– Choice of users

~ Are they a representative sample of real users?– Choice of tasks

~ Are these tasks important? To whom?– Conclusions and their relation with the experiment

~ User took time in first attempt, but this is a frequent task

~ We tested a prototype, then built the whole system, but the prototype was too small, and the system is far too complex

~ We tested the new widget, but the real problem was finding it

~ Probing confusions and evaluating task time at the same time

What can go wrong?

Reliability Validity Experimental design (mostly more validity issues)

– Testing too many things~ The UT took 4 hours, but the user was exhausted~ You cannot text everything, you have to rely on the

abilities and the experience of the designers– Confusing independent & dependent variables

~ Measuring tech-savvyness from task success

What can go wrong?

Reliability Validity Experimental design Experimenter bias

– Experimenter likes one design– The way the task is given– What the experimenter chooses to see, probe

What can go wrong?

Reliability Validity Experimental design Experimenter bias Intervening variables

– Unforeseen circumstances that might interfere with the testing or performance~ Interruptions in task time~ User took three attempts to enter amount (Rs. 500) –

this gave him extra practice with the PIN

User Testing

Gives ‘better’ results than heuristic evaluation– More confidence about findings

Needs more time and money

Task: Find 2-3 UTs of Each Type

Qualitative output Task success Task time Errors Efficiency Satisfaction scores

Think Aloud Protocol

A set of users use the product while continuously thinking out aloud– An evaluator notes all comments

Users need practice before real tasks– Use a short, familiar but unrelated task

Result is a summary of qualitative data– Excellent for formative evaluation– Can’t get performance data, particularly tasks times

Videotaping sessions helps capture nuances

Thinking Aloud Protocol Variants

Constructive Interaction– Two users use the system together

Retrospective Testing– Let users think aloud in a video-tape playback

Coaching method– Let an expert teach a novice

Think Aloud Protocol Card sort

– Louis Rosenfield, Peter Morville, Information Architecture for the World Wide Web

Categorise...

Brinjal

Camera

Cauliflower

Chicken

Electronics

Eraser

Fruits

Mobile phone

Non-veg

Notebook

Pencil

Potatoes

Raw Banana

Shampoo

Stationery

Sweet potatoes

Tomatoes

Toiletries

Toothpaste

Vegetables

Water colours

What did you use as categories? Did you make new categories?

Brinjal

Camera

Cauliflower

Chicken

Electronics

Eraser

Fruits

Mobile phone

Non-veg

Notebook

Pencil

Potatoes

Raw Banana

Shampoo

Stationery

Sweet potatoes

Tomatoes

Toiletries

Toothpaste

Vegetables

Water colours

Where did you put these?

Brinjal

Camera

Cauliflower

Chicken

Electronics

Eraser

Fruits

Mobile phone

Non-veg

Notebook

Pencil

Potatoes

Raw Banana

Shampoo

Stationery

Sweet potatoes

Tomatoes

Toiletries

Toothpaste

Vegetables

Water colours

Card Sort Technique

Make index cards– Categories– Sub-categories– Content– Number the cards (for later analysis)

Card Sort Technique

Make index cards Give a practice task

– People we know– Celebrities– Fruits and vegetables

Card Sort Technique

Make index cards Give a practice task Ask users to

– Make piles of cards– Label piles– One pile could be ‘I don’t care’ pile– Think aloud

Take notes while users work

Card Sort Variables

Open / Closed– Totally open – users write their own cards and

categories– In between – cards are given, users write category and

sub-category labels– Totally closed – pre-labelled cards and categories

– Early on – use open (discovery)– Later on – closed (validation)

Card Sort Variables

Open / Closed Phrasing

– Word, phrase, sentence~ Equipment, Having fun, Honesty is the best policy

– Category with sample sub-categories~ Equipment (Computers, Lathe machines, ACs)

– Picture, question, answer, topic, task

Card Sort Variables

Open / Closed Phrasing Granularity

– High-level or detailed– Paragraphs, pages, main pages, sub-sites– Focussed or generic– Focussed or random

Card Sort Variables

Open / Closed Phrasing Granularity Heterogeneity

– Early on – mix it up– Later on – high consistency

Card Sort Variables

Open / Closed Phrasing Granularity Heterogeneity Cross-listing

– Early on – avoid copies (structure)– Later on – allow copies (navigation)

Card Sort Variables

Open / Closed Phrasing Granularity Heterogeneity Cross-listing Qualitative / Quantitative

– Qualitative suitable for CI / mind-mapping / evaluating IA

– Quantitative suitable for summative evaluation, to do cluster analysis

Card Sort Analysis

Users’ terminologies Mind-maps

– Users’ conceptual models % times two cards go together % times a card goes in a category

Questions?

Thinking Aloud Protocol Card sort Feedback from real life usage

– Observational field studies– Logging actual use– User Feedback– Questionnaires and Interviews– Focus groups

Assignment 4

Pick a topic of your interest– Accessibility, HCI and robotics, e-learning,

Read 5 recent papers (2 years) on this topic Prepare a 10-minute presentation on this topic

– Submit a 2-page abstract with references Inform your topics to the Aakash and Abhishek by next class

– Presentations start from the class of 19-3– The presentation slots will be drawn randomly and

announced by 12-3– This assignment will be evaluated by your peers

© idc, iit bombay 1. 2 baddesigns.com © idc, iit bombay 3

usability evaluation

iit bombay task

layer of user experience

usability engineering

usability goals evaluators

unique usability problems

mix of usability experts

design formative evaluation

Documents

placements, iit bombay

communication protocols - iit bombay

iit bombay

iit bombay annual report bombay annual report 2014-2015 iit...

@iit bombay mumbai, india

iit bombay project

paper1 - iit bombay

iit bombay placement brochure - placements, iit...

aircraft - iit bombay

h13, iit bombay

research & development @ iit bombay prof. krithi ramamritham...

creativity - iit bombay

iit bombay chapter newsletter

iit bombay racing newsletter

energy paper iit bombay

idc, iit bombay - typography day 2014

wordnet-affect [iit-bombay]

iit bombay mtechbrochure2013-1418march

piping notes-iit bombay

iit bombay | iit bombay