Testing your design

• Without users:– Cognitive walkthrough– Action analysis– Heuristic analysis

• With users:– Usability testing

Messages

• Cognitive walkthrough enables a designer to evaluate an interface without users – Forces a designer to see the interface from

the perspective of a user

• Low-investment technique to identify task-related usability issues early in the design process– No implementation or users required– But can be performed on existing interfaces

Process

Site visits

Designon paper

Heuristic evaluationscognitive walkthroughs

PrototypeImplement

Field studies, call center,interaction logs

Ship

user studies

Contextualinquiry

Evaluate

User studies, function tests

Evaluate Evaluate

Evaluate

Earlydesign

Latedesign

Develop

Analysis

Analyzeusers

AnalyzeTasks

Definegoals

Cognitive Walkthrough

• Question assumptions about what the users will be thinking

• Identify controls that are obvious to the design engineer but may be hidden from the user's point of view

• Note inadequate feedback • Uncover shortcomings in the current specification • Question assumption about users tasks • Focus most clearly on a problems that users will

have when they first use an interface• A tool for developing the interface, not for

validating it

Reference

Cognitive Theory

1. The user sets a goal to be accomplished with the system (for example, "check spelling of this document").

2. The user searches the interface for currently available actions (menu items, buttons, command-line inputs, etc.).

3. The user selects the action that seems likely to make progress toward the goal.

4. The user performs the selected action and evaluates the system's feedback for evidence that progress is being made toward the current goal.

Cognitive Walkthrough

• Identify “big” problems before implementation– Invest a little now, save a lot later

• Enables more rapid iteration earlier in design– Can do several evaluations of trouble points

• Evaluations are only effective if your team– Has the right skill set– Wants to improve the design, not defend it

Walkthrough Basics

• Imagine how well a user could perform tasks with your low-fidelity prototype

• Manipulate prototype as you go– Evaluate choice-points in the interface– Evaluate labels or options– Evaluate likely user navigation errors

• Revise prototype and perform again

When to do the Walkthrough

• Have a low-fidelity prototype of the interface

• Know who the users are • Have task descriptions• Have scenarios designed to complete

the task– You have a “functional” paper prototype

• Viable once the scenario and paper prototype are completed

Who should do a walkthrough, and when?

• you can do your own, informal, “in your head” walkthroughs to monitor the design as you work

• with a group of people, including other designers and users

You need 4 things.

1. User personas– who the users will be and what kind of goals and

experience they'll bring to the job

2. A description or a low-fidelity prototype of the user interface.

3. Task descriptions 4. A complete, written list of the actions needed to

complete the task with the interface

• And an evaluation team:– Design team– Design team and users together– Design team and other skilled designers

What to do? Tell a story

• about why the user would select each action, and critique the story to make sure it's believable with these questions: 1. Will users be trying to produce whatever effect

the action has? 2. Will users see the control (button, menu,

switch, etc.) for the action? 3. Once users find the control, will they recognize

that it produces the effect they want? 4. After the action is taken, will users understand

the feedback they get, so they can go on to the next action with confidence?

More on Questions

• Some extra questions can help– What happens if the user is wrong? Is there

feedback to correct the error?– How would a user of <other interface>

react here?

• Questions help you see problems– They are a focus, not a blindfold

Best Approach

• Work as a group– don’t partition the task

• Be highly skeptical of the design – remember the goal!

• Each gap between what and how is a usability problem

Past Empirical Results

• Users will try label-guided actions first (menu items, buttons, etc.) before they experiment with direct manipulations of unlabeled objects (tools, double clicking, moving of objects).

• A well-labeled action will be especially salient. • Providing few actions in the search set can help to

narrow the search if labeling cannot be provided, or if criteria for a "good" label are difficult to establish.

• Set effects may prevent users to try untypical actions. • Users are reluctant to extend their search beyond the

readily available menus and controls. • Frequently used interfaces techniques may bias users to

search for them rather than for less frequent techniques.

Walkthrough Pros

• Easy to learn• Can perform early in the design process• Questions assumptions about what a

user may be thinking• Helps identify controls obvious to the

designer but not a user• Helps identify difficulties with labels and

prompts• Helps identify inadequate feedback

Walkthrough Cons

• Is diagnostic, not prescriptive• Focuses mostly on novice users• Designers must put themselves in the

users mind• Focus specifically on task-related issues• The interactions are slower and not real• Does not provide quantitative results

• A useful tool in conjunction with others

Walkthrough Example

• I have a library book that needs to be returned today, but I am likely to forget. To help me remember, I want to set a reminder on my laptop. The reminder should display and beep at 5:00pm to remind me to return the book to the library.

• Let’s walkthrough this task on my Outlook calendar and identify usability issues, if any

Walkthrough Example

• Will a user try to produce the effect that the action has?

• Will a user see the control for the action?• Will a user see that the control produces the

desired effect?• Will a user select a different control instead?• Will a user understand the feedback to

proceed correctly?

Your email address is invalid

joe@hahaha.com

What do you do with the results of the walkthrough?

• Fix things. – make the controls more obvious – use labels that users will recognize – eliminate actions users don't think have to

be performed or prompt for them.

Grading for Cognitive Walkthrough•

• User personas – 30 pts • Did they observe or analyze actual users doing related activities

Grade _____ (15 pts max)•   • Are the personas complete with relevant (and observed) personal details and goals

Grade _____ (15 pts max)•   

• Interface Description – 30 pts• Is there a description of a conceptual model?• Grade _____ (15 pts max)

• A description or a low-fidelity prototype of the user interface. – 15 pts • Is the description detailed enough to evaluate cues, affordances, and feedback at each decision

point Grade _____ (15 pts max)

•  

• Task descriptions – 40 pts • Are they concrete, detailed examples of all (or most) of the activities the system should

support? • Grade _____ (20pts max)•   • A complete, written list of the actions needed to complete the task with the interface – 25 pts. • Do you have a step by step story of how each persona would accomplish each task with the

proposed interface Grade _____ (20pts max)

•  

Formal action analysis

• GOMS approach (Goals Operators Methods Selection rules) – users apply specific selection rules for choosing between

available methods in the pursuit of goals. Methods consist of operators.

– The method breaks tasks in to smaller and smaller sub-goals until they can be accomplished with methods and operators of known durations. “keystroke-level analysis.”

– Can be used to predict time it takes a skilled user to complete tasks.

• Time consuming and difficult to do. • The analysis depends on the analysts. • Focuses on detail and expert performance.

Back-of-the-Envelope Action Analysis

• List the actions and think about them. • List actions including mental actions • This will highlight long and difficult

operations.

Heuristic Analysis

• Have several evaluators use the nine heuristics to identify problems with the interface, analyzing either a prototype or a paper description of the design.

• Each evaluator should do the analysis alone.

• Then combine the problems identified by the individual evaluators into a single list. Heuristic Analysis Reference - Jakob Nielsen

Nielsen’s Ten Heuristics

Visibility of system status The system should always keep users informed about what is going

on, through appropriate feedback within reasonable time.

Match between system and the real world The system should speak the users' language, with words, phrases

and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.

User control and freedom Users often choose system functions by mistake and will need a

clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

Consistency and standards Users should not have to wonder whether different words, situations,

or actions mean the same thing. Follow platform conventions.

Error prevention Even better than good error messages is a careful design which

prevents a problem from occurring in the first place.

Nielsen’s Ten Heuristics

Recognition rather than recall Make objects, actions, and options visible. The user should not have to

remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the

interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.

Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed.

Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

Help users recognize, diagnose, and recover from errors Error messages should be expressed in plain language (no codes), precisely

indicate the problem, and constructively suggest a solution.

Help and documentation Even though it is better if the system can be used without documentation, it

may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

Evaluation ResultsThe Need for Multiple Evaluators

Each row represents one of the 19 evaluators and each column represents one of the 16 usability problems. Each square shows whether the evaluator represented by the row

found the usability problem represented by the column: The square is black if this is the case and white if the evaluator did not find the problem. The rows have been sorted in

such a way that the most successful evaluators are at the bottom and the least successful are at the top. The columns have been sorted in such a way that the usability problems that are the easiest to find are to the right and the usability problems that are

the most difficult to find are to the left.