hci460: week 1 lecture september 9, 2009. 2 © august 21, 2015 course overview overview of...

HCI460: Week 1 LectureHCI460: Week 1 LectureSeptember 9, 2009September 9, 2009

2© April 19, 2023

Course overview

Overview of usability evaluation methods

Heuristic evaluation

Cognitive walkthrough

Expert evaluation

Presenting results

Project 1a

Outline

3© April 19, 2023

Course Overview

4© April 19, 2023

Instructors– Gavin Lew– Aga Bojko

Office hours– Wed 5pm – 5:45pm and 9pm – 9:45pm– Lewis 1111, Loop Campus (our classroom)

Email address– [email protected]

Course Web page– http://www.usercentric.com/hci460-fall2009.html

Prerequisites– HCI 440 and basic statistics

Basic InformationCourse Overview

5© April 19, 2023

This course surveys methods for evaluating usability of a wide range of products and interfaces.

We will discuss and practice methods:– Heuristic and expert evaluations– Cognitive walkthroughs– Usability testing (formative and

summative)– Surveys– Eye tracking– Contextual inquiries– KLM-GOMS– Focus groups

Course SummaryCourse Overview

6© April 19, 2023

To learn how to:– Establish appropriate evaluation objectives – Select evaluation methods that address evaluation objectives

and take into account existing constraints – Articulate advantages and disadvantages of usability evaluation

methods – Properly use various usability evaluation methods – Present results and prepare effective reports

Course GoalsCourse Overview

7© April 19, 2023

Required Text– Handbook of Usability Testing by Rubin

• 1st edition: ISBN 0-471-59403-2• 2nd edition: ISBN 0-470-18548-1

– Task-Centered User Interface Design: A Practical Introduction by Lewis and Rieman (online text)

Optional Text– Measuring the User Experience:

Collecting, Analyzing, and Presenting Usability Metrics by Tullis and Albert (ISBN 0-123-73558-0)

– Usability Inspection Methods by Nielsen and Mack (ISBN 0-471-01877-5)

– Various papers

TextbooksCourse Overview

8© April 19, 2023

Project 1: Expert evaluation– Individual notes– Report

Project 2: Formative usability study– Test plan– Participant screening questionnaire and moderator’s guide– Conducting a test– Report

Project 3: Quantitative comparison study– Test plan– Report

ProjectsCourse Overview

9© April 19, 2023

15% Project 1: Expert evaluation 25% Project 2: Formative usability study 15% Project 3: Quantitative comparison study 10% Take-home midterm quiz 25% Final exam 10% Individual contribution to projects (next slide)

Attendance?– Not required but strongly recommended– Projects and exams will cover both lecture and reading material

GradingCourse Overview

10© April 19, 2023

Grading: Individual Contribution to ProjectsCourse Overview

11© April 19, 2023

ScheduleCourse Overview

Date TopicText Reading

(will be assigned a week prior to each class)

Exam or Due Project

Sep 9Course overview, overview of usability evaluation methods, heuristic evaluation, cognitive walkthrough, expert evaluation

Sep 16Expert evaluation continued, fundamentals of study design

Rubin ch. 1, Lewis and Rieman ch. 4 (except 4.2)

Project 1a - Expert evaluation: individual notes

Sep 23 Formative usability testing: How to prepare a study Project 1b - Expert evaluation: final report

Sep 30 Formative usability testing: How to conduct a study Project 2a - Formative usability study: test plan

Oct 7Formative usability testing: How to analyze findings, formulate recommendations, and write reports

Project 2b - Formative usability study: participant screening questionnaire and moderator's guide

Oct 14 Teams will be conducting usability testing Project 2c - Formative usability study: conducting the testTake-home midterm will be distributed

Oct 21 Summative usability testing Take-home midterm will be collected

Oct 28 Other usability evaluation methods: Eye tracking, KLM-GOMS, contextual inquiry, surveys, remote usability testing, focus groups

Project 2d - Formative usability study: reportProject 3a - Quantitative comparison study: test plan

Nov 4

Nov 11Team presentations, overview of material for final exam

Project 3b - Quantitative comparison study: report

Nov 18 Final exam

12© April 19, 2023

Overview of Usability Evaluation Methods

13© April 19, 2023

Usability Evaluation Methods in ContextOverview of Usability Evaluation Methods

Usability testing?

Participatory design?

Survey?User

Experience (UX) Methods

Usability Evaluation Methods

14© April 19, 2023

Usability Evaluation Methods in ContextOverview of Usability Evaluation Methods

User Experience

(UX) Methods

Usability Evaluation Methods

Do not involve users Involve users

Summative usability testing

Ethnographic research

Participatory design

Card sorting

Focus groups

Eye tracking

Surveys

Formative usability testing

Heuristic evaluation

Cognitive walkthrough

KLM-GOMS

15© April 19, 2023

UX Methods Involving UsersOverview of Usability Evaluation Methods

Attitude

Behavior

Qu

alit

ativ

eQ

ua

ntitative




Participatory design

Card sorting

Focus groups

Eye tracking

Surveys

16© April 19, 2023

Usability Evaluation Methods Involving UsersOverview of Usability Evaluation Methods

Attitude

Behavior

Qu

alit

ativ

eQ

ua

ntitative




Focus groups

Eye tracking

Surveys

17© April 19, 2023

Usability Evaluation Methods Without UsersOverview of Usability Evaluation Methods

Qu

alit

ativ

eQ

ua

ntitative

Heuristic EvaluationKLM-GOMS

Cognitive Walkthrough

Expert Evaluation

Usability inspection methods:

18© April 19, 2023

Involving users is expensive (time, money).

No users = “discount usability”

Usability inspection is quick, cheap, and useful.

It should be done before a usability test to “clean up” the interface from obvious issues.– If the interface is not cleaned up, the participants will be

distracted and will waste time.

Why No Users?Overview of Usability Evaluation Methods

19© April 19, 2023

Heuristic Evaluation

20© April 19, 2023

Heuristics = guidelines, principles, rules of thumb

There are many sets of usability heuristics:– Jacob Nielsen’s Heuristics – 1994 (link)– Tognazzini’s First Principles of Interaction Design – 2003 (link)– Jill Gerhardt-Powal’s Cognitive Engineering Principles – 1996

(link)– Research-Based Web Design and Usability Guidelines – 2004

(link)

What Are Heuristics?Heuristic Evaluation

21© April 19, 2023

Nielsen’s HeuristicsHeuristic Evaluation

1. Visibility of System Status

2. Match Between System and the Real World

3. User Control and Freedom

4. Consistency and Standards

5. Error Prevention

6. Recognition Rather than Recall

7. Flexibility and Efficiency of Use

8. Aesthetic and Minimalist Design

9. Error Recovery

10.Help and Documentation

22© April 19, 2023

Nielsen’s Heuristics: #1Heuristic Evaluation

1. Visibility of System Status– The system should always keep users informed about what is

going on, through appropriate feedback within reasonable time.

There is no indication of location within the application. Clicking on Map History does not display anything if the history is empty.

23© April 19, 2023


2. Match Between System and the Real World– The system should use phrases and concepts familiar to the

user. Follow real-world conventions, making information appear in a natural and logical order.

The order of the controls is incorrect. Making a selection in the dropdown depends on whether or not the Enable Auto Update is selected.

“Processing weather data” is a system-oriented term that appears when the user clicks on “Update Weather Now.”

24© April 19, 2023


3. User Control and Freedom– Users often choose system functions by mistake and will need a

clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

Processing can take a while and there is no way to cancel the action or move the box to the side.

25© April 19, 2023


4. Consistency and Standards– Users should not have to wonder whether different words,

situations, or actions mean the same thing. Follow platform conventions.

Placement of the Browse button does not follow standards. The Browse button generally appears to the right of the file textbox.

26© April 19, 2023


5. Error Prevention– Even better than good error messages is a careful design which

prevents a problem from occurring in the first place.

The text field is too long and accepts 256 digits, suggesting that the required input should be longer than five digits.

27© April 19, 2023


6. Recognition Rather than Recall– Make objects, actions, and options visible. The user should not

have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

Most of the icons are not intuitive and they are not labeled. Users have to remember what each icon means or hover over them, which negatively impacts efficiency.

28© April 19, 2023


7. Flexibility and Efficiency of Use– Accelerators -- unseen by the novice user -- may often speed up

the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

The Close is difficult to see and the clickable area associated with it is very small. Hitting the target area is difficult and may take a few tries.

The window does not close when Alt-F4 is pressed, which is inconsistent with other Windows applications.

29© April 19, 2023


8. Aesthetic and Minimalist Design– Dialogues should not contain information which is irrelevant or

rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

It is unnecessary to include the word “Map” on all buttons. It is clear that this is a list of maps and all the actions will be performed on maps.

The tray tooltip is unnecessarily long for the amount of information it conveys.

30© April 19, 2023


9. Error Recovery– Help users recognize, diagnose, and recover from errors. Error

messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

After entering an incorrect zip code, when users click on the “Update Weather Now” icon, they see an error message.

The message provides no information on what the problem is and how to fix it.

31© April 19, 2023


10.Help and Documentation– Even though it is better if the system can be used without

documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

The Help document is very long and requires lengthy scrolling to access certain sections. This specifically impacts users who are unaware of the Find feature in Notepad.

Additionally, the sections are not labeled with the established system in the Table of Contents (A, B, C, … etc.) making the document difficult to search.

32© April 19, 2023

Form groups of 3 in class.– Online students can do this

exercise individually.

Each group will get a handout with Nielsen’s 10 heuristics and screenshots of two interfaces.

With the heuristics in mind, find a few usability issues with each interface.– Assign an appropriate

heuristic to each issue.– There may be more than one

heuristic per issue.

Nielsen’s Heuristics: ExerciseHeuristic Evaluation

1.Visibility of System StatusThe system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

2.Match Between System and the Real WorldThe system should use phrases and concepts familiar to the user. Follow real-world conventions, making information appear in a natural and logical order.

3.User Control and FreedomUsers often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

4.Consistency and StandardsUsers should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

5.Error PreventionEven better than good error messages is a careful design which prevents a problem from occurring in the first place.

6.Recognition Rather than RecallMake objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

7.Flexibility and Efficiency of UseAccelerators (unseen by the novices) – may speed up the interaction for the experts such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

8.Aesthetic and Minimalist DesignDialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

9.Error RecoveryHelp users recognize, diagnose, and recover from errors. Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

2.Help and DocumentationEven though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

33© April 19, 2023

Dog age calculator– http://www.bowwow.com.au/calculator/index.asp

Nielsen’s Heuristics: Exercise – Interface 1Heuristic Evaluation

34© April 19, 2023

Currency converter:– http://www.oanda.com/convert/classic

Nielsen’s Heuristics: Exercise – Interface 2aHeuristic Evaluation

35© April 19, 2023

The new converter: Have things improved? – http://www.oanda.com/currency/converter/

Nielsen’s Heuristics: Exercise – Interface 2bHeuristic Evaluation

36© April 19, 2023

Developed by the National Cancer Institute in the US Department of Health and Human Services– Many contributors

Available on usability.gov and in a book

2009 guidelines

Web-specific– Mostly for informational Web sites

Each guideline has two ratings

Research-Based GuidelinesHeuristic Evaluation

37© April 19, 2023

Relative importance– How important is the

guideline to the overall success of a Web site?

– Based on opinions of 16 experts

Strength of evidence– Team of researchers

evaluated existing research evidence for each guideline and rated it.

Strong research support

Moderate research support

Weak research support

Strong expert opinion support

Weak expert opinion support

Research-Based Guidelines: RatingsHeuristic Evaluation

38© April 19, 2023

How to use the guidelines?

Pair them down by making a checklist of:– Top __ guidelines based on importance– Top __ guidelines based on research evidence– Most relevant guidelines to your application

• E.g., 30 guidelines related to forms

Use guidelines with 4 – 5 strength of evidence ratings to convince others who do not believe in usability.– Credibility

Research-Based Guidelines: UsageHeuristic Evaluation

39© April 19, 2023

One person cannot find all usability problems.

Different people find different usability problems.

Heuristic evaluation is most effective with multiple evaluators.– Nielsen recommends 3 – 5 evaluators:

Conducting an Evaluation: # of EvaluatorsHeuristic Evaluation

40© April 19, 2023

Each evaluator examines the interface independently (no communication with others).– He/she can use tasks/scenarios.– At least two passes:

• 1st pass: to get a feel for the flow and scope• 2nd pass: to focus on specifics

– Output: list of problems and heuristics that they violate

Evaluators meet for a debriefing session.– Discuss each issue and the violated heuristic. – Agree on the final list of issues (ensure there is no redundancy).– Brainstorm recommendations.

Each evaluator independently assigns a severity rating to each issue.

One evaluator combines all severity ratings.

Conducting an Evaluation: StepsHeuristic Evaluation

41© April 19, 2023

Severity ratings help with prioritization of issues.

Nielsen’s 0 – 4 scale:– 0: I don’t agree that this is a usability problem at all– 1: Cosmetic problem only

• Need not be fixed unless extra time is available– 2: Minor usability problem

• Fixing this should be given low priority– 3: Major usability problem

• Important to fix, so should be given high priority– 4: Usability catastrophe

• Imperative to fix this

Conducting an Evaluation: Severity RatingsHeuristic Evaluation

42© April 19, 2023

When assigning severity ratings, consider:– The frequency with which the problem occurs.

• Is it common or rare?– The impact of the problem if it occurs.

• Will it be easy or difficult for the users to overcome?– The persistence of the problem.

• Is it a one-time problem that users can overcome once they know about it or will they be repeatedly bothered by it?


43© April 19, 2023

Another scale:


HIGH A high-severity issue is a major problem that prevents users from finding, recognizing, or using key features and completing critical tasks. It is critical to resolve this issue because it blocks the use of functionality.

A medium-severity issue can be a common problem that has a negative impact on users’ overall efficiency. It often creates inconvenience for the user. It is important to resolve this issue; otherwise user confusion and frustration will result.

A low-severity issue typically does not affect user performance but causes irritation and has impact on user’s opinion of the product/company. Resolving the issue will generally improve the user experience.

MEDIUM

LOW

44© April 19, 2023

Cognitive Walkthrough

45© April 19, 2023

Walkthroughs are a formalized way of imagining people's thoughts and actions when they use an interface for the first time.

They are more structured than a heuristic evaluation.

Walkthroughs help iterate the interface; they do not validate it.

OverviewCognitive Walkthrough

46© April 19, 2023

Interface or its prototype– E.g., Weather Watcher 5.4B

Task description (representative tasks)– E.g., Set the zip code for which you

would like Weather Watcher to retrieve the weather.)

Complete, written list of the actions needed to complete the task– E.g., click on View Options menu item, click on Active City link,

type zip code, click on OK button…

User description– E.g., wide range of computer users (from novices to experts)

What’s NeededCognitive Walkthrough

47© April 19, 2023

For each action, ask these four questions:– Will users try to produce the effect the action has? – Will users see the control (button, menu item etc.) for the action? – Will users recognize that the control produces the effect? – After the action is taken, will users understand the feedback they

get, so that they know the action took place?

StepsCognitive Walkthrough

48© April 19, 2023

Example: Setting Zip Code in Weather WatcherCognitive Walkthrough

ACTIONS 1. Click on the View Options icon on the main screen.

2. Click on Active City in the menu on the left.

3. Click on the Add New Zip Code button.

Will users try to produce the effect the action has?

Will userssee the control for the action?

Will users recognize the control produces the effect?

Will users understand the feedback?

49© April 19, 2023

Example: Setting Zip Code in Weather WatcherCognitive Walkthrough

ACTIONS 4. Type the zip code. 5. Click on the OK button to confirm the entered zip code.

6. Click on the “X” in the top right corner of the Options window to exit Options.


Will userssee the control for the action?

Will users recognize the control produces the effect?

Will users understand the feedback?

50© April 19, 2023

You should start with a list of correct actions needed to complete a given task. – You can explore the interface to identify those actions BUT

that's not the walkthrough.

The walkthrough shows what the user may have trouble with rather than what the user will do when that problem arises.

Keep in MindCognitive Walkthrough

51© April 19, 2023

Perform a cognitive walkthrough of a visual shopping site:– http://www.like.com/

Users:– Experienced Internet users– Visual shopping novices

Task: – Find shoes similar to the ones

in the picture.

Actions: – To be determined by the

evaluators.

ExerciseCognitive Walkthrough

52© April 19, 2023


Will users see the control (button, menu item etc.) for the action?

Will users recognize that the control produces the effect?

Will users understand the feedback they get, so that they know the action took place?

ExerciseCognitive Walkthrough

53© April 19, 2023

The outcome is a list of potential usability issues.

Typical recommendations: – Make the controls more obvious (question 2).– Use labels that users will recognize (question 3).– Provide better feedback (question 4).

What if Question 1 was a problem?– If the user doesn't have any reason to think that an action needs

to be performed: • Eliminate the action. Let the system take care of it. • Re-order the task so that users will start doing something

that they know needs to be done, and then get prompted for the action in question.

OutcomeCognitive Walkthrough

55© April 19, 2023

Cognitive walkthrough (CW) and heuristic evaluation (HE) with all the different sets of heuristics and are just tools.

Issues with HE and CW:– They can be unnecessarily time-consuming (you know a

problem, now you need to match a heuristic).– You will see usability issues that you will not be able to describe

with heuristics or walkthrough questions.

Thus, using only these methods may not be optimal.

Solution: Consider expert evaluation.

Practically SpeakingExpert Evaluation

56© April 19, 2023

Heuristic evaluation explicitly uses a predefined set of heuristics.– It specifies which problem violates which heuristic.

Expert evaluation:– Evaluators do not use a specific set of heuristics.– Evaluators are usability experts who have internalized various

usability guidelines and seen many usability testing sessions.– They can use elements of cognitive walkthrough but the

questions are not used explicitly.

Expert Evaluation ≠ Heuristic EvaluationExpert Evaluation

58© April 19, 2023

Introduction– Objectives– Product evaluated– Target users– Context of use

Findings with severity ratings– Describe the problem and WHY this may be a problem.– Organize by section, screen, or task rather than by heuristic.

Recommendations– Could accompany each finding or be presented at the end

(especially if there is no clear one-to-one relationship between the findings and recommendations)

What to Include in the Report?Presenting Results

59April 19, 2023

Usability Issue: The Rapid Part Add feature does not recognize part numbers if dashes are not entered. The items are added to the cart but they are not recognized.Recommendation: The search functionality should be able to handle part numbers without dashes. If it can’t, instructions should say that the part number has to include dashes. Alternatively, provide three input fields, separated by dashes. Automatically advance to the next field if maximum number of characters have been entered.

Usability Issue: The Rapid Part Add gets visually lost among all the other elements on the page and may be difficult to notice.Recommendation: Put the Rapid Part Add in a shaded box to make it stand out.

Good Practice: Rapid Part Add allows users to quickly enter their shopping list if they know the part numbers of the items without having to search for the items one by one. Also, users’ own part numbers are accepted in the Rapid Part Add tool, which is helpful.

Usability Issue: There is no error check when adding parts through Rapid Part Add. If users make an error in the part number, they will not have an opportunity to correct it.Recommendation: Provide users with feedback If the part number is not recognized. H

L

M

Presenting Results

Example

61© April 19, 2023

Evaluate Free Sticky Notes 6.0– http://www.morun.net/www/downloads/freestickynotes.html

Expert Evaluation: Individual PartProject 1a

63© April 19, 2023

Approach Project 1a

Approach:– Download the software – Familiarize yourself with the software (1st pass)– Print each screen / window /menu– Go through the application screen by screen and evaluate each

using a combination of your expertise, HE and CW– Jot down potential usability issues on the screenshots (e.g., on

post-its)

Next Wednesday, bring your individual evaluations (screenshots with your notes) to class with you. You will need 2 copies:– One for you to work with during class.– One for us (please write clearly).

64© April 19, 2023

Reading for Next WeekProject 1a

Rubin, chapter 1– What makes something usable– Techniques in UX research

Lewis and Rieman, chapter 4 (except 4.2)– Evaluating the design without users

• Cognitive walkthrough• Heuristic evaluation

hci460: week 1 lecture september 9, 2009. 2 © august 21, 2015 course overview overview of...

Documents

b expert evaluation

course surveys methods

study project

usability inspection

outline slide

slide attendance

formative usability

test report project