Formative Evaluation

cs3724: HCI

Problem scenarios

summativeevaluation

Information scenarios

claims about current practice

analysis ofstakeholders,field studies

Usability specifications

Activityscenarios

Interaction scenarios

iterativeanalysis ofusability claims andre-design

metaphors,informationtechnology,HCI theory,guidelines

formativeevaluation

DESIGN

ANALYZE

PROTOTYPE & EVALUATE

Scenario-B

ased Design

Usability Engineering

Reqs Analysis

Evaluate Design

Prototype

Usability Engineering

Reqs Analysis

Evaluate

many iterations

Design

Develop

Usability Engineering

Formative evaluation

Summative evaluation

User Interface Metrics

• Ease of learning• learning time, …

• Ease of use• perf time, error rates…

• User satisfaction• surveys…

Not “user friendly”

Claims Analysis

• + advantages

• - disadvantages

• E.g. file management:

+ +

Claims Analysis

• + advantages

• - disadvantages

• E.g. file management:

+ fast learning+ spatial learning aid memory- switching between devices- long to locate a known file…

+ flexible, power+ fast for experts - Learning commands - aesthetics lacking?...

Usability Evaluation

• Analytic Methods:• Heuristic Evaluation

• Models: GOMS, Fitts Law

• Empirical Methods:• Usability Testing

– Wizard of Oz

• Controlled Experiment

Nielsen’s 10 Heuristics1. Visible status, feedback2. User control, undo, exits3. Familiar, speak user’s language4. Consistent, standards5. Recognition over recall6. Efficient, expert shortcuts7. Aesthetic, minimalist8. Prevent errors9. Error recovery10. Help, task-based

Nielsen’s 10 Heuristics1. Visible status, feedback wysiwyg

2. User control, undo, exits wizards

3. Familiar, speak user’s language Acrobat error msg

4. Consistent, standards Word, PPT

5. Recognition over recall web nav, phone menu

6. Efficient, expert shortcuts Word bold

7. Aesthetic, minimalist phone book

8. Prevent errors HomeFinder

9. Error recovery undo, back

10. Help, task-based IIS help doc

Speak the User’s Language

Help documentation

Context help, help doc, UI

Applying Heuristics

• Lets look at a website…

Early Usability Testing

• Formative: helps guide design

• Early in design process• when architecture is finalized, then its too late!

• A few users (3-5)

• Identify usability problems, incidents

• Qualitative feedback from users

• Quantitative usability measures

Usability Test Procedure• Wizard of Oz: act out the system• Benchmark task set• No cheating! Don’t teach them how to use the UI• Repeat:

• Give user a task• Ask user to “think aloud”• Observe, note mistakes and problems• Avoid interfering, hint only if completely stuck

• Interview • Verbal feedback• Questionnaire

• 30 min - 1 hour / user

Analyze Results• Initial reaction:

• “stupid user!”, “that’s developer X’s fault!”, “this sucks”

• Mature reaction:• “how can we redesign UI to solve that usability problem?”• the user is always ‘right’

• Identify usability problems• Learning issues: e.g. can’t figure out or didn’t notice feature• Performance issues: e.g. arduous, tiring to solve tasks• Subjective issues: e.g. annoying, ugly

• Problem severity: critical vs. minor• Redesign

Pizza Ordering System

• Scenario:

• Volunteers!

• Design ideas

• Wizard of Oz usability test

• Redesign

In action: Pizza Ordering System

• Scenario: VT students, studying/busy/tired, poor, wants discounts, starving, credit card / hokie card/ no cash, 2 in together, half-n-half, split cost

• Volunteers!

• Design ideas

• Wizard of Oz usability test

• Redesign

Pizza R Us

peperoni

mushrooms

pineapple

WholeHalf-n-half

PRICE: $10.00

Drag your toppings:

checkout

• Size of pizza:– large– Small

• First half:

• Toppings:– Peperoni– Mushrooms– Pineapple

• Price: 0.00

• 2nd half:

• Toppings:– Peperoni– Mushrooms– pineapple

• % pza –ppm –large -…