usability and human factors usability evaluation methods lecture b this material (comp15_unit5b) was...

Usability and Human Factors

Usability Evaluation Methods

Lecture b

This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003.

Usability Evaluation MethodsLearning Objectives

2

• Conduct a cognitive walkthrough (Lecture b)

• Design appropriate tasks for a usability test (Lecture b)

• Describe the usability testing environment, required equipment, logistics, and materials (Lecture b)

Health IT Workforce Curriculum Version 3.0/Spring 2012

Usability & Human Factors Usability Evaluation Methods

Lecture b

Cognitive Walkthrough (Polson et al, 1992)

3Health IT Workforce Curriculum Version 3.0/Spring 2012


Lecture b

• Kind of cognitive task analysis• Assesses system usability• Criteria focuses on cognitive processes

needed to perform tasks• Identifying sequences of actions and

subgoals to successfully complete a task• Assigning causes to usability problems• Are the cues provided by the interface

sufficient to perform task?

Cognitive Task Analysis (CTA)

• Tools and techniques for describing knowledge & strategies required for task performance– Hierarchical decomposition of goals and

component tasks

• Objective: – Yield information about the knowledge,

thought processes, and goal structures that underlie observable task performance



Lecture b

Why Do a CTA?

• Develop theory of competent performance• Understand invariant features of a task

– There are invariant performance characteristics of any class of tasks

• Understand process of skill acquisition• Training and instructional resources

– e.g. manuals and tutorials• Develop methods for usability testing

– Design– Coding scheme for data analysis



Lecture b

Cognitive Walkthrough Step 1: Preparations



Lecture b

Step 2: Walkthrough Process



Lecture b

Step 3: Explicate Sources of Potential

Problems



Lecture b

ATMs: Goal Structure



Lecture b

A Partial Walkthrough: ATMGoal: Obtain $80 Cash from Checking

Account



Lecture b

Cognitive Walkthrough: Measure Glucose



Lecture b

CW: Blood Glucose 2



Lecture b

CW: Blood Glucose 3



Lecture b

CW Glucose Results



Lecture b

Usability Testing

• Gold standard for usability evaluation• Set of techniques to collect empirical data

– while observing representative end users using the system under study to perform representative tasks

• Video-recorded• Provide information that can lead to systems that:

– Easy to learn and use– Satisfying to use– Provide utility and functionality that are valued by the

target population– Characterize task-specific competencies



Lecture b

Think-Aloud Protocol

• Method broadly used in cognitive research and usability testing

• User verbalizes his/her thoughts while performing a task– Report the contents of working memory– Session is audio and/or video recorded

• Transcript of think aloud is coordinated with video analysis



Lecture b

Selection of Representative Users

• Users may differ:– including age, education, gender, computer

experience, etc.

• Select subjects based on relevant criteria (e.g., age, education)

• Fully representative not possible

• Convenience sample is less desirable



Lecture b

Development of Test Plan

• Outline task and procedure– Informed by objectives– Prior testing– Constrained by time and setting– Ethical and IRB issues

• Exploratory – Characterize potential problems

• Controlled Experiment– Comparing 2 Interfaces



Lecture b

Role of Researcher

• Neutral Observer vs. Active Participant

• Researchers may play a more interactive role in field testing – Guide the subject as necessary

• Skilled user will need a minimum• Novice may need step-by-step instructions

– Autonomy Rule: No more guidance than necessary



Lecture b

Field Usability Testing

• Hybrid Method: Lab and Ethnography/ Field Study

• Naturalistic setting– Numerous constraints

• Proscribed set of tasks– Quasi- Experiment

• Video analysis is key– Intrusive



Lecture b

Video-Analytic Usability Testing on Location: Old School

Microsoft Clipart



Lecture b

Software-based Video Analysis

• Provides a video of all screen activity • Captures user via a webcam• Logs a wide range of events and system

interactions including mouse clicks, text entries, web-page changes and windows dialogue events (e.g., saving a document, selecting among a set of choices).

• Morae state of the art usability testing software



Lecture b

Morae Video Analytic Usability Software

Khan, S.A., Ancker, J.S., Li, J., Kaufman, D., Hutchinson, C., Cohall, A., Kukafka, R. (2009)



Lecture b

Data Analysis: Transcript

• Working document for video analysis

• Verbatim and Time Stamped– Every 10 to 30 seconds

• Iteratively modified document as coding categories become refined

• Add field notes and observations to the transcript



Lecture b

Video Analysis: Granularity



Lecture b

Macro-AnalysisMarch 18, 2002

I: … You can choose not to answer any questions. May I ask how old are you?

P: 74

I: What is or was your professional occupation.

P: I have always worked in stores as a salesperson

I: How long have you lived in this country?

P: 48, 49 years

I: If I may ask what level of education do you have?

P: I finished high school, but in Colombia. Before I got married, then I came here after I got married, a couple of years afterwards.

I: How long have you known that you have diabetes.

P: A couple of years. About two years or three. I have never felt that I have diabetes.

Profile:

74 years of age

Female

Native Spanish speaker

Originally from Colombia

48-49 years in US

High school education

Worked as salesperson in stores

Has two sons, one in NJ

Never used computer prior to IDEATel

Diabetes History:

Had it for 2-3 years

Did not have monitoring device prior to IDEATel

1y.2months in program

Good health/Has asthma

A lot of fluctuation in glucose values

Does not look at them over a period of time, watches it one day to the other.

Generally good memory remembers values for previous days

1.1 Table: (Kaufman, et al., 2003).



Lecture b

Observations: Sending Results



Lecture b

Micro-AnalysisTime

Dialogue and Conversational Code

Action Code System Response

Comments

7:28 R: Can you explain what you see now?P: It is calling the Internet. There it turns off, and leaves me with the system.

Body Position Change (BPC): Patient moves head, looks from side to sideGesture: Points at screen with index fingerAction: Takes mouse and clicks security screen awayBPC: keeps hand on mouse for a couple of seconds, then offAction: Takes mouse and clicks identification screen away.

Security screen/dialogue box appearsSec. Screen goes awayPassword screen appears

The task is to access the web and patient immediately assumes the control

7:54 R: So far, so good. Action: Takes hand off mouse, hunches over keyboard, and puts in password.Action: Takes mouse and clicks OK button and hand off mouse.Action: Takes mouse in hand immediately as DM page comes up.

Password screen transitions to Diabetes Manager Page

Patient performs all actions without any prompting.

1.2 Table: (Kaufman, et al., 2003).



Lecture b

Triangulate



Lecture b

Usability Evaluation Methods Summary

• The value of usability evaluations in healthcare contexts has been well established

• Wide range of methods which vary in terms of their advantages and disadvantages

• Lectures illustrated how to use someof these methods including the cognitive walkthrough and usability testing



Lecture b

Usability Evaluation MethodsReferences – Lecture b

References

1. Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. John Wiley & Sons, New York, NY.

2. Polson, P., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741–773.

3. Preece, J., Rogers, Y., & Sharp, H. (2007). Interaction Design: Beyond Human-Computer Interaction (2nd ed.). West Sussex, England: Wiley.

Images

Slide 21: Microsoft Clipart

Slide 23: Khan, S.A., Ancker, J.S., Li, J., Kaufman, D., Hutchinson, C., Cohall, A., Kukafka, R. (2009). GetHealthyHarlem.org: developing a web platform for health promotion and wellness driven by and for the Harlem community. AMIA Annu Symp Proc.317–21.

Charts, Tables & Figures

1.1 Table: Kaufman, D.R., Patel, V.L., Hilliman, C., Morin, P.C., Pevzner, J, Weinstock, Goland, R. Shea, S. & Starren, J. (2003). Usability in the real world: Assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics, 36, 45-60.

1.2 Table: Kaufman, D.R., Patel, V.L., Hilliman, C., Morin, P.C., Pevzner, J, Weinstock, Goland, R. Shea, S. & Starren, J. (2003). Usability in the real world: Assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics, 36, 45-60.



Lecture b

usability and human factors usability evaluation methods lecture b this material (comp15_unit5b) was...

Documents

usability problemsare

workforce curriculum

usability testing environment

human services

tutorialsdevelop methods

department of health

materials lecture bhealth

health information technology