Evaluation Paradigms & Techniques

IS 588

Spring 2008

Dr. D. Bilal

Overview

• Evaluation is performed to determine how well a certain product design meets user needs.

• Need to decide what to evaluate?– Guided by goals, theory, model, etc.

• What to evaluate determines how to do the evaluation

Evaluation Paradigms

• Quick & Dirty

• Usability testing

• Field studies

• Predictive evaluation

Quick & Dirty

• Informal

• Designers or evaluators meet informally with users– Gather information about product design– Gather suggestions for design improvements

• Inexpensive

• Not time consuming

Usability Testing

• Formal assessment• Measures user performance on predefined tasks

– Tasks structured based on purpose of evaluation

• Controlled by evaluator• Performance observed and/or captured

– EXAMPLES?– based on questions guiding usability testing (i.e., what

the evaluator wants to find)

Usability Testing

• Typically quantitative

• Interviews and questionnaires can result in qualitative assessments – User comments, quotes of likes/dislikes, etc.

• A mix method is ideal– WHY?

Usability Testing

• Not performed in a naturalistic setting

• Activities can be captured/recorded using software (e.g., Morae, HyperCam, Camtasia), or videotape

• Evaluator may take observational notes while activities being captured

Field Studies

• Naturalistic setting – User interacts with system as part of a daily

routine– No tasks given by evaluator– Evaluator observes and records activities, OR

uses software to capture activities, OR…

• Can be qualitative and quantitative– HOW?

Predictive Evaluation

• Experts place themselves in the users’ shoes to predict usability problems

• Guided by heuristics– Quick, inexpensive– Limitations

• WHAT ARE THEY?

Evaluation Techniques

• Observe user

• Gather user opinion

• Gather expert opinion

• Test user performance

• Model user performance

• Mix method (2 or more techniques)

DECIDE Framework

• Determine goals

• Explore/set questions to be answered

• Choose suitable paradigms and techniques

• Identify issues (e.g., how to recruit participants)

DECIDE Framework

• Decide on tackling ethical concerns (e.g., use of human subjects, privacy)

• Evaluate, interpret, present data