Lecture 7:Lecture 7:User-centered approaches and User-centered approaches and

Introducing evaluationIntroducing evaluation

OverviewOverview

•Why involve users at all?•What is a user-centered approach?•Understanding users’ work

—Coherence—Contextual Design

•Involving users in design—PICTIVE—CARD

Why involve users at all?Why involve users at all? •Expectation management

• Realistic expectations • No surprises, no disappointments• Timely training• Communication, but no hype

•Ownership • Make the users active stakeholders• More likely to forgive or accept problems• Can make a big difference to acceptance and success of product

Degrees of user involvement Degrees of user involvement •Member of the design team

— Full time: constant input, but lose touch with users— Part time: patchy input, and very stressful— Short term: inconsistent across project life— Long term: consistent, but lose touch with users

•Newsletters and other dissemination devices — Reach wider selection of users— Need communication both ways

•Combination of these approaches

How Microsoft involves usersHow Microsoft involves users

Users are involved throughout development •‘activity-based planning’: studying what users do to achieve a certain activity (task) •usability tests e.g. Office 4.0 over 8000 hours of usability testing.•internal use by Microsoft staff•customer support lines

User-centered approach is based on:User-centered approach is based on:• Early focus on users and tasks: directly studying Early focus on users and tasks: directly studying

cognitive, behavioral, anthropomorphic & attitudinal cognitive, behavioral, anthropomorphic & attitudinal characteristics characteristics

• Empirical measurement:Empirical measurement: users’ reactions and users’ reactions and performance to scenarios, manuals, simulations & performance to scenarios, manuals, simulations & prototypes are observed, recorded and analysedprototypes are observed, recorded and analysed

• Iterative designIterative design:: when problems are found in user when problems are found in user testing, fix them and carry out more teststesting, fix them and carry out more tests

What is a user-centered approach?What is a user-centered approach?

Early focus on users and tasksEarly focus on users and tasks Users’ tasks and goals are the driving force behind the Users’ tasks and goals are the driving force behind the

development development

Users’ behavior and context of use are studied and the Users’ behavior and context of use are studied and the product is designed to support them product is designed to support them

Users’ characteristics are captured & designed for Users’ characteristics are captured & designed for

Users are consulted throughout development, from Users are consulted throughout development, from earliest phases to the latest, and their input is seriously earliest phases to the latest, and their input is seriously taken into account taken into account

All design decisions are taken within the context of the All design decisions are taken within the context of the user, their work and their environment user, their work and their environment

Understanding users’ work Understanding users’ work

•Understanding users’ work is significant

•Ethnography: from anthropology

•‘writing the culture’ and participant observation•the branch of anthropology that provides scientific description of

individual human societies

•That branch of knowledge which has for its subject the

characteristics of the human family, developing the details with which

ethnology as a comparative science deals; descriptive ethnology.

•Difficult to use the output of ethnography in

design

Framework for using ethnography in designFramework for using ethnography in design

•Distributed co-ordination: distributed nature of the tasks & activities, and the means and mechanisms by which they are co-ordinated

•Plans and procedures: organisational support for the work, such as workflow models and organisational charts, and how these are used to support the work

•Awareness of work: how people keep themselves aware of others’ work

Maslow's Hierarchy of Needs Maslow's Hierarchy of Needs 1) Physiological: hunger, thirst, bodily comforts, etc.; 1) Physiological: hunger, thirst, bodily comforts, etc.; 2) Safety/security: out of danger; 2) Safety/security: out of danger; 3) Belonginess and Love: affiliate with others, be 3) Belonginess and Love: affiliate with others, be

accepted; and accepted; and 4) Esteem: to achieve, be competent, gain approval 4) Esteem: to achieve, be competent, gain approval

and recognition.and recognition. 5) Cognitive: to know, to understand, and explore; 5) Cognitive: to know, to understand, and explore; 6) Aesthetic: symmetry, order, and beauty; 6) Aesthetic: symmetry, order, and beauty; 7) 7) Self-actualizationSelf-actualization: to find self-fulfillment and realize : to find self-fulfillment and realize

one's potential; and one's potential; and 8) 8) Self-transcendenceSelf-transcendence: to connect to something : to connect to something

beyond the ego or to help others find self-fulfillment beyond the ego or to help others find self-fulfillment and realize their potential.and realize their potential.

CoherenceCoherence

•A method which offers appropriate questions to help address these key dimensions

•For example:

—Distributed Coordination: How is the division of labor manifest through the work of individuals and its co-ordination with others?

—Plans and procedures: How do plans and procedures function in the workplace?

Contextual DesignContextual Design•Developed to handle data collection and analysis from fieldwork for developing a software-based product

•Used quite widely commercially

•Contextual Design has seven parts: Contextual inquiry, Work modelling, Consolidation, Work redesign, User environment design,Mock-up and test with customers, Putting it into Practice

Contextual InquiryContextual Inquiry

•An approach to ethnographic study where user is expert, designer is apprentice•A form of interview, but

—at users’ workplace (workstation)—2 to 3 hours long

•Four main principles:—Context: see workplace & what happens—Partnership: user and developer collaborate—Interpretation: observations interpreted by user and developer together—Focus: project focus to help understand what to look for

Work ModelingWork ModelingIn interpretation session, models are drawn from the observations:

•Work flow model: the people, communication and co-ordination

•Sequence model: detailed work steps to achieve a goal

•Artifact model: the physical ‘things’ created to do the work

•Cultural model: constraints on the system from organizational culture

•Physical model: physical structure of the work, e.g. office layout

ConsolidationConsolidation

•Each contextual inquiry (one for each user/developer pair) results in a set of models•These need to be consolidated into one view of ‘the work’•Affinity diagram

—Organizes interpretation session notes into common structures and themes

—Categories arise from the data —Diagram is built through induction

•Work models consolidated into one of each type

Participatory DesignParticipatory Design

•Scandinavian history

•Emphasises social and organisational aspects

•Based on study, model-building and analysis

of new and potential future systems

Participatory Design (contd)Participatory Design (contd)•Aspects to user involvement include

—Who will represent the user community?Interaction may need to be assisted by a facilitator—Shared representations—Co-design using simple tools such as paper or video scenarios—Designers and users communicate about proposed designs—Cooperative evaluation such as assessment of prototypes

Benefits of Participatory DesignBenefits of Participatory Design

“Computer-based systems that are poorly suited to how people actually work impose cost not only on the

organisation in terms of low productivity but also on the people who work with them. Studies of work in

computer-intensive workplaces have pointed to a host of serious problems that can be caused by job design

that is insensitive to the nature of the work being performed, or to the needs of human beings in an

automated workplace.”

[Kuhn, S. in Bringing Design to Software, 1996]

PICTIVEPICTIVE

•Plastic Interface for Collaborative Technology Initiatives through Video Exploration

•Intended to empower users to act a full participants in design

PICTIVE (contd)PICTIVE (contd)

•Materials used are:—Low-fidelity office items such as pens, paper, sticky notes—Collection of (plastic) design objects for screen and window layouts

•Equipment required:—Shared design surface, e.g. table —Video recording equipment

PICTIVE (contd)PICTIVE (contd)•Before a PICTIVE session:

—Users generate scenarios of use —Developers produce design elements for the design session

•A PICTIVE session has four parts:—Stakeholders all introduce themselves—Brief tutorials about areas represented in the session (optional)—Brainstorming of ideas for the design—Walkthrough of the design and summary of decisions made

CARCARDD

•Collaborative Analysis of Requirements & Design •Similar to PICTIVE but at a higher level of abstraction: explores work flow not detailed screen design•Uses playing cards with pictures of computers and screen dumps•Similar structure to the session as for PICTIVE•PICTIVE and CARD can be used together to give complementary views of a design

SummarySummary• User involvement helps manage users’ expectations & User involvement helps manage users’ expectations &

feelings of ownershipfeelings of ownership

• A user-centered approach has three main elements: early A user-centered approach has three main elements: early focus on users, empirical measurement and iterative focus on users, empirical measurement and iterative designdesign

• Ethnography is useful for understanding work, but can be Ethnography is useful for understanding work, but can be difficult to use in designdifficult to use in design

• Coherence and Contextual Design support the use of Coherence and Contextual Design support the use of ethnographic data in designethnographic data in design

• Participative design involves users taking an active part in Participative design involves users taking an active part in design decisionsdesign decisions

• CARD and PICTIVE are example techniquesCARD and PICTIVE are example techniques

Introducing evaluationIntroducing evaluation

The aimsThe aims

Discuss how developers cope with real-Discuss how developers cope with real-world constraints.world constraints.

Explain the concepts and terms used to Explain the concepts and terms used to discuss evaluation. discuss evaluation.

Examine how different techniques are Examine how different techniques are used at different stages of development.used at different stages of development.

Two main types of evaluationTwo main types of evaluation

Formative evaluationFormative evaluation is done at is done at different stages of development to check different stages of development to check that the product meets users’ needs.that the product meets users’ needs.

Summative evaluationSummative evaluation assesses the assesses the quality of a finished product. quality of a finished product.

Our focus is on formative evaluationOur focus is on formative evaluation

Iterative design & evaluation is a Iterative design & evaluation is a continuous process that examines:continuous process that examines:• Early ideas for conceptual model Early ideas for conceptual model • Early prototypes of the new systemEarly prototypes of the new system• Later, more complete prototypesLater, more complete prototypes

Designers need to check that they Designers need to check that they understand users’ requirements.understand users’ requirements.

What to evaluateWhat to evaluate

Bruce Tognazzini tells you why you Bruce Tognazzini tells you why you need to evaluateneed to evaluate

““Iterative design, with its repeating cycle of Iterative design, with its repeating cycle of design and testing, is the only validated design and testing, is the only validated methodology in existence that will consistently methodology in existence that will consistently produce successful results. If you don’t have user-produce successful results. If you don’t have user-testing as an integral part of your design process testing as an integral part of your design process you are going to throw buckets of money down you are going to throw buckets of money down the drain.” the drain.”

See AskTog.com for topical discussion about See AskTog.com for topical discussion about design and evaluation.design and evaluation.

When to evaluateWhen to evaluate

Throughout designThroughout design From the first descriptions, sketches From the first descriptions, sketches

etc. of users needs through to the etc. of users needs through to the final productfinal product

Design proceeds through iterative Design proceeds through iterative cycles of ‘design-test-redesign’cycles of ‘design-test-redesign’

Evaluation is a key ingredient for a Evaluation is a key ingredient for a successful design.successful design.

Evaluating the 1984 OMSEvaluating the 1984 OMS Early tests of printed scenarios & user guidesEarly tests of printed scenarios & user guides Early simulations of telephone keypad Early simulations of telephone keypad An Olympian joined team to provide feedbackAn Olympian joined team to provide feedback Interviews & demos with Olympians outside USInterviews & demos with Olympians outside US Overseas interface tests with friends and family.Overseas interface tests with friends and family. Free coffee and donut testsFree coffee and donut tests Usability tests with 100 participants.Usability tests with 100 participants. A ‘try to destroy it’ test A ‘try to destroy it’ test Pre-Olympic field-test at an international eventPre-Olympic field-test at an international event Reliability of the system with heavy trafficReliability of the system with heavy traffic

Development of HutchWorldDevelopment of HutchWorld Many informal meetings with patients, carers & Many informal meetings with patients, carers &

medical staff early in designmedical staff early in design

Early prototype was informally tested on siteEarly prototype was informally tested on site Designers learned a lot e.g.Designers learned a lot e.g.

- language of designers & users was different - language of designers & users was different - asynchronous communication was also - asynchronous communication was also neededneeded

Redesigned to produce the portal versionRedesigned to produce the portal version

Usability testingUsability testing

User tasks investigated:User tasks investigated:- how users’ identify was represented- how users’ identify was represented- communication- communication- information searching- information searching- entertainment- entertainment

User satisfaction questionnaireUser satisfaction questionnaire Triangulation to get different Triangulation to get different

perspectivesperspectives

Findings from the usability testFindings from the usability test

The back button didn’t always work The back button didn’t always work Users didn’t pay attention to navigation Users didn’t pay attention to navigation

buttonsbuttons Users expected all objects in the 3-D view to Users expected all objects in the 3-D view to

be clickable.be clickable. Users did not realize that there could be Users did not realize that there could be

others in the 3-D world with whom to chat, others in the 3-D world with whom to chat, Users tried to chat to the participant list. Users tried to chat to the participant list.

Key pointsKey points

Evaluation & design are closely integrated in user-Evaluation & design are closely integrated in user-centered design.centered design.

Some of the same techniques are used in Some of the same techniques are used in evaluation & requirements but they are used evaluation & requirements but they are used differently (e.g., interviews & questionnaires)differently (e.g., interviews & questionnaires)

Triangulation involves using a combination of Triangulation involves using a combination of techniques to gain different perspectivestechniques to gain different perspectivesDealing with constraints is an important skill for Dealing with constraints is an important skill for evaluators to develop. evaluators to develop.