1 cmt 4240 week 7 qualitative evaluation methods
Post on 19-Dec-2015
220 views
TRANSCRIPT
1
CMT 4240 week 7
Qualitative Evaluation Methods
2
Objectives
At the end of this lecture you will be able to carry out and analyse the following evaluation methods: Observation verbal protocols cognitive walkthrough
3
Observation
Direct observationrecorded observationsoftware logging
4
Direct observation
Observation in the real world and in the lab researcher makes notes of user behaviourquestions:
where do you position yourself? What do you watch the screen or the user? which user behaviour is relevant? How fast and how much can you write? Do you have to note conversation literally?
5
Individual Exercise Observation
Imagine you are in the science museumYou evaluate the prototype “secrets
behind your screen”The design team of the museum has
asked you to find out how long on average visitors use the exhibit problems that visitors experience when using
the exhibit
6
Exercise Observation
What do you do (step by step)?Which problems do you expect?What are you going to take with you?5 minutes
7
What are you going to take with you?
Notepad and penscardboard with clip(stop-) watchobservation forms...
8
Potential Problems
Where to position yourself? Need to see the user’s actions and his non-verbal behaviour.
User may feel disturbed, if you watch to closely.How to stop the time and to observe potential
usability problems at the same time.Hawthorne effect
9
Potential Problems
Only one pass at the data collection observer has to decide in real time what is
important and what is not observer may miss important data observer may misconceive events or
misunderstand the user observer has to know from the outset what to
look out for
10
Potential Problems
You may get distractedYour handwriting may be illegibleYou may not remember what you meant
with certain wordsYou may jot down too few words
11
So what do you do?
Preparation: make observation sheets
start time, end timeuser age (approximately) and genderactionsproblems
12
What do you do?
Position sideways between exhibit and user so that you can see both
Write down everything as explicitly as possible
conversations need to be written down as literal as possible
Keep to the observable FACTS not your interpretation of them
13
Recorded observation
Advantage: you can repeatedly look at the recordings and
discover new things every time
Disadvantage: technology is fine, if it works... You need electricity, extension leads, several
cameras, tapes, etc. it can be costly
14
Analysing Observation Material
Recordings: make transcripts transcripts and notes should be coded Informal analysis
what are the problems?
Task based analysis how does the user tackle a given task?
Performance based analysis obtain data on predefined performance measures
15
Verbal protocols
Verbal protocols capture the user’s thoughts and spoken observations
Two types of verbal protocols think aloud protocol post event protocol
verbal protocols are often used in combination with video recording
16
Think aloud protocol
The user is being asked to say out loud what he or she think while he or she is carrying out a task
17
Post event protocol
The user is being asked to think out loud after the task has been completed
18
Exercise Think Aloud Protocol:
Get together in pairsOne student is the observer and one
student is the subjectThe observer makes notesThe subject solves the following problem
while thinking out loud: how does the following series continue:
0, 1/3, 2/4, 3/5, 4/6, ?
19
Solution?
20
Analysis of the protocol
Go through your notes and ask the following questions:
How many different attempts to solve the problem can you identify?
How many different problems did your subject encounter?
Which statement in the protocol supports which problems solving activity and which problem encoutered?
21
Cognitive Walkthrough
Evaluation method based on the idea that… the user learns about the interface in an
exploratory way, the user has goals, the user is applying simple means-ends
reasoning.
22
Cognitive Walkthrough
Evaluation by an expert, who goes through a set task while imitating user performance
23
Preparation: We need four things
1. User description including level of experience with computers and any assumptions made by the designer
2. System description including operations and performance (e.g. paper design)
3. Task description specifying the task that the expert has to carry out from users point of view
4. Action sequence describing the system display and the user actions needed to complete the given task. One system display and one user action together are one step.
24
Process of carrying out a Cognitive Walkthrough
Prototype, user description, system description, task description and action list has to be prepared by the designer/developer
Designer gives these document to an expert.The expert reads the descriptions.The expert carries out the task by following the
action list.The expert asks the following questions with
EACH step of the action list:
25
Questions Cognitive Walkthrough
1. Is the next goal clear at this stage?
2. Is the appropriate action obvious?
3. Is it clear that this action leads to the goal?
4. What problems are there in performing the action?
26
Example: the Automatic Nail Painter
The automatic nail painter is a machine with which one can have one’s nails painted.
27
User Description Automatic Nail Painter
The users are assumed to have no prior knowledge of how to operate the Automatic Nail Painter. However, they are assumed to have experience in putting nail varnish on nails manually
28
System Description Automatic Nail Painter
The system allows one finger to be placed into the machine at a time. The user programmes the machine for use by choosing:- the number of fingers to be varnished- the colours of the varnish to be used- the number of coatings to be applied
29
Task Description Automatic Nail Painter
The user must get all ten fingers varnished with one coat of purple varnish. The system is already switched on when the user begins his/her task. The task is completed, when the tenth fingernail is varnished.
30
Action list Automatic Nail Painter
System: System displays ‘How many fingers do you want to varnish?’User action: user selects number 10 by pressing the + or the - button until the appropriate number is shownSystem: System responds by displaying the current number of fingers selected to be varnished.User action: User presses ok when the correct number has been displayed
31
Action list Automatic Nail Painter (cont.)
System: System displays ‘choose the colour of varnish you wish to apply.’User action: User must select the colour purple by choosing from the colour menu. This includes scrolling up and down until s/he finds purple. S/he then has to press the OK button.System: Systems displays ‘how many coats do you want to apply?’User action: User selects the number of coats by pressing the ‘+’ or ‘-’ button until the 1 is shown.
32
Action list Automatic Nail Painter (cont.)
System: System displays the number of fingers to be varnished, the colour of the varnish and the number of coats to be applied. System asks for confirmationUser action: User presses OK to confirmSystem: System displays ‘Press start to begin varnishing.’User action: User presses start buttonetc.
33
Step 1- Question 1 Automatic Nail Painter
From the action list: Step 1System: System displays ‘How many fingers do you
want to varnish?’User action: user selects number 10 by pressing the + or the - button until the appropriate number is shown
Question 1:Is the goal clear at this stage? Not completely.The user is asked to state the number of fingers to be varnished. He could be thinking that the system deals with one hand at a time.
34
Step 1 - Questions 2, 3, and 4Automatic Nail Painter
Question 2:Is the appropriate action obvious? No.The system dos neither indicate that the user has to use the ‘+’ and ‘-’ buttons nor explain their use.
Question 3:Is it clear that this action leads to the goal? YesIf the user presses the ‘+’ and ’-’ button the numbers are clearly displayed.
Question 4: What problems are there in performing the action?
The action is simple to perform once the user has discovered the use of the buttons.
35
Step 2 – Question 1Automatic Nailpainter
?
36
Summary
Direct and recorded ObservationVerbal Protocol
Think aloud protocol
Cognitive Walkthrough Preparation: System description, user
description, task description, Action sequence Process: for each step in the action sequence
four questions to be answered