ICS 463, Intro to Human Computer Interaction Design:

8. Evaluation and Data

Dan Suthers

Evaluation

• Formative: early and often; informs design

• Summative: near end; have we done well?

• We focus on formative

When and why?

• Early in the life cycle– understanding target application– check understanding of requirements – quick filtering of ideas

• Middle– predicting usability– comparing alternate designs– engineering towards a usability target

• Later– fine tuning of usability – verifying conformance to a standard

Preview of Methods of Evaluation

• Collecting users' opinions attitudes

• Observing and monitoring use how users interact

• Experiments hypothesis testing

• Interpretive evaluation how used in natural settings (ecological

validity)• Predictive evaluation

anticipated usability issues

Typical Procedure (formal evals)

• Identify questions • Plan the evaluation• Pilot the evaluation and revise if needed

• Run the sessions and collect the data• Analyze the data • Draw conclusions • Redesign and revise system• Details next week (Hix & Hartson)

Dimensions to consider

Evaluation planning should consider • Characteristics of users • Types of activities • Environment of use • Nature of artifact

So should design … from the outset!

Data

• Types of Data– Objective versus Subjective – Quantitative versus Qualitative

• What are the independent variables?

• Dependent variables? • Controlled variables?Now on to details of evaluation

methods …

Collecting Users’ Opinions

Tells us about attitudesCaveat: "First rule of usability: don't

listen to users, watch what they do!"

Two major methods• Interviews - qualitative analysis• Surveys - quantitative analysis

Interviews

• Structured: – fixed questions, fixed or conditional

sequence – easier to conduct and analyze– may miss opportunistic information

• Semi-structured– set of questions “to get to”

• Flexible– no set questions or sequence

Questionnaires and Surveys

• Large numbers, analyzed quantitatively• Design with your analysis in mind • Piloting important• Closed questions versus open questions • Types of closed questions

– Checklists: background information– Likert scales: range of agreement or

disagreement with a statement– Semantic differentials: place on scale of

adjectives – Ranked order: e.g., rank in order of

usefulness

Observing Users

• Observing and monitoring use of artifact – in laboratory– in natural setting how users interact with system usability issues

Direct Observation

• Researcher watches use, takes notes

• Hawthorne Effect (users act differently under observation) may contaminate results

• Record may be incomplete • Only one chance • Helpful to have shorthand and/or

forms which which you are fluent

Indirect Observation

• Video logging – User(s) body language, gestures– Screen activity

• Two uses: – Exploratory evaluation: review tapes

carefully and repeatedly to discover issues – Formal studies: know what you are looking

for! • Interaction logging (software)

– Often use two or more together – Must synchronize all data streams

• High volume of data can be overwhelming

Verbal Protocols

• Audio record of spoken language– Spontaneous utterances – Conversation between multiple users– Think-aloud protocol– Post-event protocols

• Dangers of introspection, rationalization

• Analyze along with video

Video/Verbal Analysis

• Diversity of approaches • Task-based

– how do users approach the problem– difficulties in using the software– need not be exhaustive: identify

interesting cases • Performance-based

– frequency and timing of categories of actions, errors, task completion

More on Analysis of Video/Verbal

• Requires classification scheme, invented or borrowed

• May involve inter-rater reliability• Often exhaustive and time

intensive! • Tools important

– we transcribe conversation to text merged with transaction log

– better approach would be direct analysis of digital video

Software Instrumentation/Logging

• Time-stamped logs – key-presses or higher level actions– record what happened but not

replayable

• Interaction logging – replayable

• Synchronize with video data for rich but overwhelming data

• Analysis issues are similar

Methods of Evaluation

• Collecting users' opinions attitudes

• Observing and monitoring use of artifact – in laboratory– in natural setting how users interact

• Experiments – testable hypothesis– comparative with controlled variables– quantitative analysis

• Interpretive evaluation – how used in natural settings – qualitative data and analysis

• Predictive evaluation – predict usability issues based on model– applied to specifications or prototypes

Additional Methods

• Experiments – testable hypothesis– comparative with controlled variables– quantitative analysis – we’ll get into this next week

• Interpretive evaluation – how used in natural settings – qualitative data and analysis

• Predictive evaluation – predict usability issues based on model– applied to specifications or prototypes

Assignment 7 (project groups)

• The good news: your project has been accepted by The Boss!

• The bad news: you have 6 weeks to finish it!

Assignment 7 continued

• Write a 2 page plan expressed in terms of the Star model– What is the expected product? – At which phase will you start? – What is your estimated timeline? – How will you incorporate evaluation and other

user-centered techniques, and use the outcomes to adjust the design?

– What resources do you need? – This can be in outline form and will be

assessed for its value as an efficient briefing for The Boss