evaluation, cont’d
DESCRIPTION
Evaluation, cont’d. Two main types of evaluation. Formative evaluation is done at different stages of development to check that the product meets users’ needs. Summative evaluation assesses the quality of a finished product. Our focus is on formative evaluation. What to evaluate. - PowerPoint PPT PresentationTRANSCRIPT
Evaluation, cont’d
Two main types of evaluation
Formative evaluation is done at different stages of development to check that the product meets users’ needs.
Summative evaluation assesses the quality of a finished product.
Our focus is on formative evaluation
Iterative design & evaluation is a continuous process that examines:
– Early ideas for conceptual model – Early prototypes of the new system– Later, more complete prototypes
Designers need to check that they understand users’ requirements.
What to evaluate
Tog says …
“Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain.”
When to evaluate
Throughout design From the first descriptions, sketches
etc. of users needs through to the final product
Design proceeds through iterative cycles of ‘design-test-redesign’
Evaluation is a key ingredient for a successful design.
Another example - development of “HutchWorld”
Many informal meetings with patients, carers & medical staff early in design
Early prototype informally tested on site– Designers learned a lot
• language of designers & users was different• asynchronous communication was also needed
Redesigned to produce the portal version
Usability testing
– User tasks investigated:- how users’ identify was represented- communication- information searching- entertainment
User satisfaction questionnaire
Triangulation to get different perspectives
Findings from the usability test
• The back button didn’t always work
• Users didn’t pay attention to navigation buttons
• Users expected all objects in the 3-D view to be clickable.
• Users did not realize that there could be others in the 3-D world with whom to chat,
• Users tried to chat to the participant list.
Key points Evaluation & design are closely integrated in
user-centered design. Some of the same techniques are used in
evaluation & requirements but they are used differently (e.g., interviews & questionnaires)
Triangulation involves using a combination of techniques to gain different perspectives
Dealing with constraints is an important skill for evaluators to develop.
A case in point …
“The Butterfly Ballot: Anatomy of disaster”.See http://www.asktog.com/columns/042ButterflyBallot.html
An evaluation framework
The aims
Explain key evaluation concepts & terms.Describe the evaluation paradigms &
techniques used in interaction design.Discuss the conceptual, practical and
ethical issues that must be considered when planning evaluations.
Introduce the DECIDE framework.
Evaluation paradigm
Any kind of evaluation is guided explicitly or implicitly by a set of beliefs, which are often under-pinned by theory. These beliefs and the methods associated with them are known as an ‘evaluation paradigm’
User studies
User studies involve looking at how people behave in their natural environments, or in the laboratory, both with old technologies and with new ones.
Four evaluation paradigms
‘quick and dirty’ usability testing field studies predictive evaluation
Quick and dirty
‘quick & dirty’ evaluation describes the common practice in which designers informally get feedback from users or consultants to confirm that their ideas are in-line with users’ needs and are liked.
Quick & dirty evaluations are done any time. The emphasis is on fast input to the design
process rather than carefully documented findings.
Usability testing Usability testing involves recording typical users’
performance on typical tasks in controlled settings. Field observations may also be used.
As the users perform these tasks they are watched & recorded on video & their key presses are logged.
This data is used to calculate performance times, identify errors & help explain why the users did what they did.
User satisfaction questionnaires & interviews are used to elicit users’ opinions.
Field studies
Field studies are done in natural settings The aim is to understand what users do
naturally and how technology impacts them. In product design field studies can be used
to:- identify opportunities for new technology- determine design requirements - decide how best to introduce new technology- evaluate technology in use.
Predictive evaluation
Experts apply their knowledge of typical users, often guided by heuristics, to predict usability problems.
Another approach involves theoretically based models.
A key feature of predictive evaluation is that users need not be present
Relatively quick & inexpensive
Overview of techniques
observing users, asking users’ their opinions, asking experts’ their opinions, testing users’ performance modeling users’ task performance
DECIDE: A framework to guide evaluation Determine the goals the evaluation addresses. Explore the specific questions to be answered. Choose the evaluation paradigm and
techniques to answer the questions. Identify the practical issues. Decide how to deal with the ethical issues. Evaluate, interpret and present the data.
Determine the goals
What are the high-level goals of the evaluation?
Who wants it and why?
The goals influence the paradigm for the study
Some examples of goals: Identify the best metaphor on which to base the design. Check to ensure that the final interface is consistent. Investigate how technology affects working practices. Improve the usability of an existing product .
Explore the questions
All evaluations need goals & questions to guide them so time is not wasted on ill-defined studies.
For example, the goal of finding out why many customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions:- What are customers’ attitudes to these new tickets? - Are they concerned about security?- Is the interface for obtaining them poor?
What questions might you ask about the design of a cell phone?
Choose the evaluation paradigm & techniques The evaluation paradigm strongly
influences the techniques used, how data is analyzed and presented.
E.g. field studies do not involve testing or modeling
Identify practical issues
For example, how to:
• select users
• stay on budget
• staying on schedule
• find evaluators
• select equipment
Decide on ethical issues
Develop an informed consent form
Participants have a right to:- know the goals of the study- know what will happen to the findings- privacy of personal information- not to be quoted without their agreement - leave when they wish - be treated politely
Evaluate, interpret & present data
How data is analyzed & presented depends on the paradigm and techniques used.
The following also need to be considered:- Reliability: can the study be replicated?- Validity: is it measuring what you thought?- Biases: is the process creating biases?- Scope: can the findings be generalized?- Ecological validity: is the environment of the study influencing it - e.g. Hawthorne effect
Pilot studies
A small trial run of the main study. The aim is to make sure your plan is viable. Pilot studies check:
- that you can conduct the procedure- that interview scripts, questionnaires, experiments, etc. work appropriately
It’s worth doing several to iron out problems before doing the main study.
Ask colleagues if you can’t spare real users.
Key points An evaluation paradigm is an approach that is influenced by
particular theories and philosophies.
Five categories of techniques were identified: observing users, asking users, asking experts, user testing, modeling users.
The DECIDE framework has six parts: - Determine the overall goals- Explore the questions that satisfy the goals- Choose the paradigm and techniques- Identify the practical issues- Decide on the ethical issues- Evaluate ways to analyze & present data
Do a pilot study
Observing users
The aims Discuss the benefits & challenges of different
types of observation. Describe how to observe as an on-looker, a
participant, & an ethnographer. Discuss how to collect, analyze & present
observational data. Examine think-aloud, diary studies & logging. Provide you with experience in doing observation
and critiquing observation studies.
What and when to observe
Goals & questions determine the paradigms and techniques used.
Observation is valuable any time during design. Quick & dirty observations early in design Observation can be done in the field (i.e., field
studies) and in controlled environments (i.e., usability studies)
Observers can be:- outsiders looking on- participants, i.e., participant observers- ethnographers
Frameworks to guide observation
- The person. Who? - The place. Where?- The thing. What?
The Goetz and LeCompte (1984) framework:- Who is present? - What is their role? - What is happening? - When does the activity occur?- Where is it happening? - Why is it happening? - How is the activity organized?
The Robinson (1993) framework
Space. What is the physical space like? Actors. Who is involved? Activities. What are they doing? Objects. What objects are present? Acts. What are individuals doing? Events. What kind of event is it? Goals. What do they to accomplish? Feelings. What is the mood of the group and of
individuals?
You need to consider
Goals & questions Which framework & techniques How to collect data Which equipment to use How to gain acceptance How to handle sensitive issues Whether and how to involve informants How to analyze the data Whether to triangulate
Observing as an outsider
As in usability testing More objective than participant observation In usability lab equipment is in place Recording is continuous Analysis & observation almost simultaneous Care needed to avoid drowning in data Analysis can be coarse or fine grained Video clips can be powerful for telling story
Participant observation & ethnography Debate about differences Participant observation is key component of
ethnography Must get co-operation of people observed Informants are useful Data analysis is continuous Interpretivist technique Questions get refined as understanding grows Reports usually contain examples
Data collection techniques
Notes & still camera Audio & still camera Video Tracking users:
- diaries- interaction logging
Data analysis
Qualitative data - interpreted & used to tell the ‘story’ about what was observed.
Qualitative data - categorized using techniques such as content analysis.
Quantitative data - collected from interaction & video logs. Presented as values, tables, charts, graphs and treated statistically.
Interpretive data analysis
Look for key events that drive the group’s activity Look for patterns of behavior Test data sources against each other - triangulate Report findings in a convincing and honest way Produce ‘rich’ or ‘thick descriptions’ Include quotes, pictures, and anecdotes Software tools can be useful e.g., NUDIST,
Ethnograph (URLs will be provided)
Looking for patterns
Critical incident analysis Content analysis Discourse analysis Quantitative analysis - i.e., statistics
Key points
Observe from outside or as a participant Analyzing video and data logs can be time-
consuming. In participant observation collections of
comments, incidents, and artifacts are made. Ethnography is a philosophy with a set of techniques that include participant observation and interviews.
Ethnographers immerse themselves in the culture that they study.
Asking users & experts
The aims
Discuss the role of interviews & questionnaires in evaluation.
Teach basic questionnaire design. Describe how do interviews, heuristic
evaluation & walkthroughs. Describe how to collect, analyze &
present data. Discuss strengths & limitations of these
techniques
Interviews
Unstructured - are not directed by a script. Rich but not replicable.
Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness.
Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability.
Basics of interviewing
Remember the DECIDE framework Goals and questions guide all interviews Two types of questions:
‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’‘open questions’ do not have a predetermined format
Closed questions are quicker and easier to analyze
Things to avoid when preparing interview questions
Long questions Compound sentences - split into two Jargon & language that the interviewee may
not understand Leading questions that make assumptions
e.g., why do you like …? Unconscious biases e.g., gender stereotypes
Components of an interview
Introduction - introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present an informed consent form.
Warm-up - make first questions easy & non-threatening.
Main body – present questions in a logical order
A cool-off period - include a few easy questions to defuse tension at the end
Closure - thank interviewee, signal the end, e.g, switch recorder off.
The interview process
Use the DECIDE framework for guidance Dress in a similar way to participants Check recording equipment in advance Devise a system for coding names of
participants to preserve confidentiality. Be pleasant Ask participants to complete an informed
consent form
Probes and prompts
Probes - devices for getting more information.e.g., ‘would you like to add anything?’
Prompts - devices to help interviewee, e.g., help with remembering a name
Remember that probing and prompting should not create bias.
Too much can encourage participants to try to guess the answer.
Group interviews Also known as ‘focus groups’ Typically 3-10 participants Provide a diverse range of opinions Need to be managed to:
- ensure everyone contributes- discussion isn’t dominated by one person- the agenda of topics is covered
Analyzing interview data Depends on the type of interview Structured interviews can be
analyzed like questionnaires Unstructured interviews generate
data like that from participant observation
It is best to analyze unstructured interviews as soon as possible to identify topics and themes from the data
Questionnaires Questions can be closed or open Closed questions are easiest to analyze, and
may be done by computer Can be administered to large populations Paper, email & the web used for
dissemination Advantage of electronic questionnaires is that
data goes into a data base & is easy to analyze
Sampling can be a problem when the size of a population is unknown as is common online
Questionnaire style Varies according to goal so use the DECIDE
framework for guidance Questionnaire format can include:
- ‘yes’, ‘no’ checkboxes- checkboxes that offer many options- Likert rating scales- semantic scales- open-ended responses
Likert scales have a range of points 3, 5, 7 & 9 point scales are common Debate about which is best
Developing a questionnaire Provide a clear statement of purpose &
guarantee participants anonymity Plan questions - if developing a web-based
questionnaire, design off-line first Decide on whether phrases will all be
positive, all negative or mixed Pilot test questions - are they clear, is there
sufficient space for responses Decide how data will be analyzed & consult a
statistician if necessary
Encouraging a good response
Make sure purpose of study is clear Promise anonymity Ensure questionnaire is well designed Offer a short version for those who do not
have time to complete a long questionnaire If mailed, include a s.a.e. Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is high, 20% is often
acceptable
Advantages of online questionnaires
Responses are usually received quickly No copying and postage costs Data can be collected in database for
analysis Time required for data analysis is reduced Errors can be corrected easily Disadvantage - sampling problematic if
population size unknown Disadvantage - preventing individuals from
responding more than once
Problems with online questionnaires
Sampling is problematic if population size is unknown
Preventing individuals from responding more than once
Individuals have also been known to change questions in email questionnaires
Questionnaire data analysis & presentation
Present results clearly - tables may help Simple statistics can say a lot, e.g., mean,
median, mode, standard deviation Percentages are useful but give population
size Bar graphs show categorical data well More advanced statistics can be used if
needed
Well-known forms
SUMI MUMMS QUIS -- see Perlman site
Asking experts
Experts use their knowledge of users & technology to review software usability
Expert critiques (crits) can be formal or informal reports
Heuristic evaluation is a review guided by a set of heuristics
Walkthroughs involve stepping through a pre-planned scenario noting potential problems
Heuristic evaluation
Developed by Jakob Nielsen in the early 1990s
Based on heuristics distilled from an empirical analysis of 249 usability problems
These heuristics have been revised for current technology, e.g., HOMERUN for web
Heuristics still needed for mobile devices, wearables, virtual worlds, etc.
Design guidelines form a basis for developing heuristics
Nielsen’s heuristics Visibility of system status Match between system and real world User control and freedom Consistency and standards Help users recognize, diagnose, recover from
errors Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help and documentation
Discount evaluation
Heuristic evaluation is referred to as discount evaluation when 5 evaluators are used.
Empirical evidence suggests that on average 5 evaluators identify 75-80% of usability problems.
3 stages for doing heuristic evaluation
Briefing session to tell experts what to do Evaluation period of 1-2 hours in which:
- Each expert works separately- Take one pass to get a feel for the product- Take a second pass to focus on specific features
Debriefing session in which experts work together to prioritize problems
Advantages and problems
Few ethical & practical issues to consider Can be difficult & expensive to find
experts Best experts have knowledge of
application domain & users Biggest problems
- important problems may get missed- many trivial problems are often identified
Cognitive walkthroughs
Focus on ease of learning Designer presents an aspect of the
design & usage scenarios One of more experts walk through the
design prototype with the scenario Expert is told the assumptions about
user population, context of use, task details
Experts are guided by 3 questions
The 3 questions Will the correct action be sufficiently evident
to the user? Will the user notice that the correct action is
available? Will the user associate and interpret the
response from the action correctly?
As the experts work through the scenario they note problems
Pluralistic walkthrough Variation on the cognitive walkthrough theme Performed by a carefully managed team The panel of experts begins by working
separately Then there is managed discussion that leads
to agreed decisions The approach lends itself well to participatory
design
Key points Structured, unstructured, semi-structured
interviews, focus groups & questionnaires Closed questions are easiest to analyze & can
be replicated Open questions are richer Check boxes, Likert & semantic scales Expert evaluation: heuristic & walkthroughs Relatively inexpensive because no users Heuristic evaluation relatively easy to learn May miss key problems & identify false ones
A project for you …
Activeworlds.com Questionnaire to test reactions with friends http://
www.acm.org/~perlman/question.html http://www.ifsm.umbc.edu/djenni1/osg/ Develop heuristics to evaluate usability
and sociability aspects
A project for you …
http://www.id-book.com/catherb/
-provides heuristics and a template so that you can evaluate different kinds of systems. More information about this is provided in the interactivities section of
the id-book.com website.
A project for you …
Go to the The Pew Internet & American Life Survey www.pewinternet.org/ (or to another survey of your choice)
Critique one of the recent online surveys
Critique a recent survey report
Interpretive Evaluation
Contextual inquiry Cooperative and participative evaluation Ethnography
rather than emphasizing statement of goals, objective tests, research reports, instead emphasizes usefulness of findings to the people concerned
good for feasibility study, design feedback, post-implementation review
Contextual Inquiry
Users and researchers participate to identify and understand usability problems within the normal working environment of the user
Differences from other methods include:– work context -- larger tasks– time context -- longer times– motivational context -- more user control– social context -- social support included that is
normally lacking in experiments
Why use contextual inquiry?
Usability issues located that go undetected in laboratory testing.– Line counting in word processing– unpacking and setting up equipment
Issues identified by users or by user/evaluator
Contextual interview: topics of interest Structure and language used in work individual and group actions and
intentions culture affecting the work explicit and implicit aspects of the work
Cooperative evaluation
A technique to improve a user interface specification by detecting the possible usability problems in an early prototype or partial simulation
low cost, little training needed think aloud protocols collected during
evaluation
Cooperative Evaluation
Typical user(s) recruited representative tasks selected user verbalizes problems/ evaluator
makes notes debriefing sessions held Summarize and report back to design
team
Participative Evaluation
More open than cooperative evaluation subject to greater control by users cooperative prototyping, facilitated by
– focus groups– designers work with users to prepare
prototypes– stable prototypes provided, users evaluate– tight feedback loop with designers
Ethnography
Standard practice in anthropology Researchers strive to immerse themselves in
the situation they want to learn about Goal: understand the ‘real’ work situation typically applies video - videos viewed,
reviewed, logged, analyzed, collections made, often placed in databases, retrieved, visualized ….
Predictive Evaluation
Predict aspects of usage rather than observe and measure
doesn’t involve users cheaper
Predictive Evaluation Methods
Inspection Methods– Standards inspections– Consistency inspection– Heuristic evaluation– “Discount” usability evaluation– Walkthroughs
Modelling:The keystroke level model
Standards inspections
Standards experts inspect the interface for compliance with specified standards
relatively little task knowledge required
Consistency inspections
Teams of designers inspect a set of interfaces for a family of products– usually one designer from each project
Usage simulations
Aka - “expert review”, “expert simulation”
Experts simulate behavior of less-experienced users, try to anticipate usability problems
more efficient than user trials prescriptive feedback
Heuristic evaluation
Usage simulation in which system is evaluted against list of “heuristics”, e.g.
Two passes: per screen, and flow from screeen to screen
Study: 5 evaluators found 75% of problems
Sample heuristics
Use simple and natural dialogue speak the user’s language minimize user memory load be consistent provide feedback provide clearly marked exits provide shortcuts provide good error messages prevent errors
Discount usability engineering
Phase 1: usability testing + scenario construction (1-3 users)
Phase 2: scenarios refined + heuristic evaluation
“Discount” features– small scenarios, paper mockups– informal think-aloud (no psychologists)– Scenarios + think-aloud + heuristic evaluation– small number of heuristics (see previous slide)– 2-3 testers sufficient
Walkthroughs
Goal - detect problems early on; remove construct carefully designed tasks from a
system specification or screen mockup walk-through the activities required, predict
how users would likely behave, determine problems they will encounter
-- see checklist for cognitive walkthrough
Modeling: keystroke level model
Goal: calculate task performance times for experienced users
Requires– specification of system functionality– task analysis, breakdown of each task into
its components
Keystroke-level modeling
Time to execute sum of:– Tk - keystroking (0.35 sec)– Tp - pointing (1.10)– Td - drawing (problem-
dependent)– Tm - mental (1.35)– Th - homing (0.4)– Tr - system response (1.2)
KLM: example
Save file with new name in wp that uses mouse and pulldown menus
(1) initial homing: (Th) (2) move cursor to file menu at top of screen(Tp + Tm
) (3) select ‘save as’ in file menu(click on file menu,
move down file menu, click on ‘save as’) (Tm + Tk + Tp +Tk)
(4) word processor prompts for new file name, user types filename (Tr + Tm + Tk(filename) + Tk)
Experiments and Benchmarking
Traditional experiments Usability Engineering
Traditional Experiments
Typically narrowly defined, evaluate particular aspects such as:– menu depth v. context– icon design– tickers v. fade_boxes v. replace_boxes
Usually not practical to include in design process
Example: Star Workstation,text selection Goal: evaluate methods for selecting
text, using 1-3 mouse buttons Operations:
– Point (between characters, target of move,copy, or insert)
– Select text (character, word, sentence, par, doc)
– Extend selection to include more text
Selection Schemes
A B C D E F G
Button1
Point Point Point
C
Drwthru
Point
C, W, S, P, D
Drwthru
Point
C, W, S, P, D, Drwthru
Point
C
Dthru
Point
C, W, S, P, D
Button2
C
Drwthru
C, W, S, P, D
Drwthru
W, S, P, D
Drwthru
Adjust Adjust Adjust
Button3
W, S, P, D
Drwthru
Methodology
Between-subjects paradigm six groups, 4 subjects per group in each group: 2 experienced w/mouse, 2 not each subject first trained in use of mouse and
in editing techniques in Star w.p. system Assigned scheme taught Each subject performs 10 text-editing tasks, 6
times each
Results: selection time
Time:
Scheme A :12.25 sScheme B: 15.19 sScheme C: 13.41 sScheme D: 13.44 sScheme E: 12.85 sScheme F: 9.89 s (p < 0.001)
Results: Selection Errors
Average: 1 selection error per four tasks 65% of errors were drawthrough errrors,
same across all selection schemes 20% of errors were “too many clicks” ,
schemes with less clicking better 15% of errors were ‘click wrong mouse
button”, schemes with fewer buttons better
Selection scheme: test 2
Results of test 1 lead to conclusion to avoid:– drawthroughs– three buttons– multiple clicking
Scheme “G” introduced -- avoids drawthrough, uses only 2 buttons
New test, but test groups were 3:1 experienced w/mouse to not
Results of test 2
Mean selection time: 7.96s for scheme G, frequency of “too many clicks” stayed about the same
Conclusion: scheme G acceptable– selection time shorter– advantage of quick selection balances
moderate error rate of multi-clicking
Experimental design - concerns
What to change? What to keep constant? What to measure?
Hypothesis, stated in a way that can be tested.
Statistical tests: which ones, why?
Variables
Independent variable - the one the experimenter manipulates (input)
Dependent variable - affected by the independent varialbe (output)
experimental effect - changes in dependent caused by changes in independent
confounded -- when dependent changes because of other variables (task order, learning, fatigue, etc.)
Selecting subjects - avoiding bias
Age bias -- Cover target age range Gender bias -- equal numbers of
male/female Experience bias -- similar level of
experience with computers etc. ...
Experimental Designs
Independent subject design– single group of subjects allocated randomly to each of the
experimental conditions
Matched subject design– subjects matched in pairs, pairs allocated randomly to each
of the experimental conditions
Repeated measures design– all subjects appear in all experimental conditions– Concerns: order of tasks, learning effects
Single subject design– in-depth experiments on just one subject
Critical review of experimental procedure User preparation
– adequate instructions and training? Impact of variables
– how do changes in independent variables affect users
Structure of the tasks– were tasks complex enough, did users know aim?
Time taken– fatigue or boredom?
Critical review of experimental results Size of effect
– statistically signficant? Practically significant? Alternative interpretations
– other possible causes for results found? Consistency between dependent variables
– task completion and error scores versus user preferences and learning scores
Generalization of results– to other tasks,users, working environments?
Usability Engineering
Usability of product specified quantitatively, and in advance
As product is built, it can be demonstrated that it does or does not reach required levels of usability
Usability Engineering
Define usability goals through metrics Set planned levels of usability that need to
be achieved Analyze the impact of various design
solutions Incorporate user-defined feedback in
product design Iterate through design-evaluate-design
loop until planned levels are achieved
Metrics
Include:– time to complete a particular task– number of errors– attitude ratings by users
Metrics - example, conferencing system
Attribute Measuring
ConceptMeasuring
Method
Worst case Planned level Best case Now level
Initial use Conferencing task
successful interxns / 30 min
1-2 3-4 8-10 ?
Infreq. Use Tasks after 1-2 weeks disuse
% of errors Equal to product Z
50% better 0 errors ?
Learning rate Task 1st half vs. 2nd half score
Two halves equal
Second half better
‘much’ better ?
Preference over prod. Z
Questionnaire
Score
Ratio of scores
Same as Z None prefer Z ?
Pref over product A
Questionnaire
Score
Ratio of scores
Same as Q None prefer Q
?
Error recovery
Critical incident analysis
% incidents accounted for
10% 50% 100% ?
Initial evaluation
Attitude questionnaire
Semantic differential score
0 (neutral) 1(somewhat positive)
2 (highly positive)
?
Casual eval. Attitude questionnaire
Semantic differential score
0 (neutral) 1(somewhat positive)
2 (highly positive)
?
Mastery eval Attitude questionnaire
Semantic differential score
0 (neutral) 1(somewhat positive)
2 (highly positive)
?
Benchmark tasks
Carefully constructed standard tests used to monitor users’ performance in usability testing
typically use multiple videos, keyboard logging
controlled testing -- specified set of users, well-specified tasks, controlled environment
tasks longer than scientific experiments, shorter than “real life”
Making tradeoffs
impact analysis - used to establish priorities among usability attributes. It is a listing of attributes and proposed design decisions, and % impact of each.
Usability engineering reported to produce a measurable improvement in usability of about 30%.