observation watch, listen, and learn…. agenda observation exercise come back at 3:40. questions?...
Post on 19-Dec-2015
223 views
TRANSCRIPT
![Page 1: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/1.jpg)
Observation
Watch, listen, and learn…
![Page 2: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/2.jpg)
Agenda
Observation exercise Come back at 3:40.
Questions? Observation
![Page 3: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/3.jpg)
Observing Users
Not as easy as you think
One of the best ways to gather feedback about your interface
Watch, listen and learn as a person interacts with your system
Qualitative & quantitative, end users, experimental or naturalistic
![Page 4: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/4.jpg)
Observation
Direct In same room Can be intrusive Users aware of
your presence Only see it one
time May use 1-way
mirror to reduce intrusiveness
IndirectVideo or app recordingReduces intrusiveness, but doesn’t eliminate itCameras focused on screen, face & keyboardGives archival record, but can spend a lot of time reviewing it
![Page 5: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/5.jpg)
Location
Observations may be In lab - Maybe a specially built usability
lab Easier to control Can have user complete set of tasks
In field Watch their everyday actions More realistic Harder to control other factors
![Page 6: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/6.jpg)
UsabilityLab
http://www.surgeworks.com/services/observation_room2.htm
Large viewing area in this one-way mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms.
Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room.
![Page 7: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/7.jpg)
ObservationRoom
State-of-the-art observation room equipped with three monitors to view participant, participant's monitor, and composite picture in picture.
One-way mirror plus angled glass captures light and isolates sound between rooms.
Comfortable and spacious for three people, but room enough for six seated observers.
Digital mixer for unlimited mixing of input images and recording to VHS, SVHS, or MiniDV recorders.
![Page 8: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/8.jpg)
Task Selection
What tasks are people performing? Representative and realistic? Tasks dealing with specific parts of the
interface you want to test? Problematic tasks?
Don’t forget to pilot your entire evaluation!!
![Page 9: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/9.jpg)
Engaging Users in Evaluation
What’s going on in the user’s head? Use verbal protocol where users describe
their thoughts
Qualitative techniques Think-aloud - can be very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions
“What did you like best/least?” “How would you change..?”
![Page 10: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/10.jpg)
Think Aloud
User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do
Widely used, popular protocol Potential problems:
Can be awkward for participant Thinking aloud can modify way user performs
task
![Page 11: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/11.jpg)
Cooperative approach
Another technique: Co-discovery learning (Constructive iteration) Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert
(coach) and one be novice More natural (like conversation) so removes
some awkwardness of individual think aloud Variant: let coach be from design team
(cooperative evaluation)
![Page 12: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/12.jpg)
Alternative
What if thinking aloud during session will be too disruptive?
Can use post-event protocol User performs session, then watches
video afterwards and describes what s/he was thinking
Sometimes difficult to recall Opens up door of interpretation
![Page 13: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/13.jpg)
What if a user gets stuck?
Decide ahead of time what you will do. Offer assistance or not? What kind of
assistance?
You can ask (in cooperative evaluation) “What are you trying to do..?” “What made you think..?” “How would you like to perform..?” “What would make this easier to accomplish..?” Maybe offer hints This is why cooperative approaches are used
![Page 14: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/14.jpg)
Inputs / Outcomes
Need operational prototype could use Wizard of Oz simulation
What you get out “process” or “how-to” information Errors, problems with the interface compare user’s (verbalized) mental
model to designer’s intended model
![Page 15: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/15.jpg)
Capturing a Session
1. Paper & pencil Can be slow May miss things Is definitely cheap and easy
Time 10:00 10:03 10:08 10:22
Task 1 Task 2 Task 3 …
Se
Se
![Page 16: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/16.jpg)
Capturing a Session
2. Recording (audio and/or video) Good for think-aloud Hard to tie to interface Multiple cameras may be needed Good, rich record of session Can be intrusive Can be painful to transcribe and
analyze
![Page 17: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/17.jpg)
Capturing a Session
3. Software logging Modify software to log user actions Can give time-stamped key press or
mouse event Two problems:
Too low-level, want higher level events Massive amount of data, need analysis
tools
![Page 18: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/18.jpg)
Example logs2303761098721869683|hrichter|1098722080134|MV|START|5662303761098721869683|hrichter|1098722122205|MV|QUESTION|false|false|false|false|false|false| 2303761098721869683|hrichter|1098724978982|MV|TAB|AGENDA2303761098721869683|hrichter|1098724981146|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098724985161|MV|SLIDECHANGE|52303761098721869683|hrichter|1098724986904|MV|SEEK|PRESENTATION-A|566|604189|02303761098721869683|hrichter|1098724996257|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098724998791|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098725002506|MV|TAB|AGENDA2303761098721869683|hrichter|1098725003848|MV|SEEK|AGENDA|566|149613|6041892303761098721869683|hrichter|1098725005981|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725007133|MV|SLIDECHANGE|32303761098721869683|hrichter|1098725009326|MV|SEEK|PRESENTATION|566|315796|1496132303761098721869683|hrichter|1098725011569|MV|PLAY|566|3157962303761098721869683|hrichter|1098725039850|MV|TAB|AV2303761098721869683|hrichter|1098725054241|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725056053|MV|SLIDECHANGE|22303761098721869683|hrichter|1098725057365|MV|SEEK|PRESENTATION|566|271191|3157962303761098721869683|hrichter|1098725064986|MV|TAB|AV2303761098721869683|hrichter|1098725083373|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725084534|MV|TAB|AGENDA2303761098721869683|hrichter|1098725085255|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725088690|MV|TAB|AV2303761098721869683|hrichter|1098725130500|MV|TAB|AGENDA2303761098721869683|hrichter|1098725139643|MV|TAB|AV2303761098721869683|hrichter|1098726430039|MV|STOP|566|2711912303761098721869683|hrichter|1098726432482|MV|END
![Page 19: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/19.jpg)
Analysis
Many approaches Task based
How do users approach the problem What problems do users have Need not be exhaustive, look for interesting
cases Performance based
Frequency and timing of actions, errors, task completion, etc.
Very time consuming!!
![Page 20: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/20.jpg)
Example: Heather’s study
Software: MeetingViewer interface fully functional
Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing
Resources – subjects were students in a research group, just me as evaluator, plenty of time
Wanted completely authentic experience
![Page 21: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/21.jpg)
Heather’s evaluation
Task: answer questions from a recorded meeting, use my software as desired
Think-aloud Video taped, software logs Also had post questionnaire Wrote my own code for log analysis Watched video and matched behavior to
software logs
![Page 22: Observation Watch, listen, and learn…. Agenda Observation exercise Come back at 3:40. Questions? Observation](https://reader036.vdocuments.site/reader036/viewer/2022062304/56649d3f5503460f94a18bca/html5/thumbnails/22.jpg)
Example: Analysis
Printed logs Watched video and marked actions,
thoughts, and video timing info on the logs
Wrote down interesting quotes from the think aloud
Took 2 to 3 times the session time just to do this analysis alone!