Hvad er Remote Usability Testing?

Jan Stage

Professor, PhD

Forskningsleder i Informationssystemer (IS)/Human-Computer Interaction (HCI)

Aalborg Universitet, Institut for Datalogi, HCI-Lab

jans@cs.aau.dk

Institut for Datalogi 2

• Forhindringer i praksis

• Ideen om remote usability testing

• Realisering af ideen

Oversigt

Institut for Datalogi 3

• Forhindringer i praksis

• Ideen om remote usability testing

• Realisering af ideen

Oversigt

Institut for Datalogi 4

To undersøgelser af forhindringer i praksis

Primært Nordjylland

1. Software-virksom-heder, typisk svar fra mellemledere

2. Deltagere i en-dages konference om usability

• Tallene angiver procent af svarene i hver undersøgelse

2325

15

86

13

0

10

43

118

5 5

1

26

00

5

10

15

20

25

30

35

40

45

50

Resourcedemands

Developermindset

Customerparticipation

Testparticipants

Conductingtests

No problems Low priority No answ er

Første undersøgelse

Anden undersøgelse

Institut for Datalogi 5

• Forhindringer i praksis

• Ideen om remote usability testing

• Definition og baggrund

• Tidligere arbejde

• Synkrone og asynkrone metoder

• Realisering af ideen

Oversigt

Institut for Datalogi 6

Definition and Motivation

• Remote usability testing is a situation where“the evaluators are separated in space and/or time from users”(Castillo, Hartson and Hix, 1998)

• Well-known challenges• Building and setting up usability labs• Getting users to the lab and making them feel comfortable

• New challenges• Users, developers and evaluators are distributed geographically, sometimes

even in different countries and time zones• Outsourcing• Open Source Software development• Global software development

• Remote usability evaluation is becoming increasingly relevant

Institut for Datalogi 7

Earlier Work (1)

• The idea of remote evaluation emerged between 1994 and 1998

• Literature survey on remote usability testing• 21 references• Early: presents a variety of methods• Recently: practical advice and experiences combined with pros/cons

Institut for Datalogi

Earlier Work (2)

8

Institut for Datalogi 9

Earlier Work (3)

• Two types of methods: synchronous / asynchronous (various forms)

• The majority of references focus on synchronous usability evaluation:Simulating a conventional laboratory-based think-aloud test

• Overall:• Few systematic comparisons of remote methods• Mostly comparisons of a single remote method and the conventional lab-based

method

Institut for Datalogi 10

Synchronous Methods

Laboratory testing (LAB)• Conventional method• Benchmark for comparison

Remote synchronous testing (RS)• Spatial separation: User and test monitor in

separate rooms• Remote sharing of desktop• Webcam connection with picture in the

bottom corner of the screen• Audio connection

Data: Video recording of the test monitor’s screen and the audio

Institut for Datalogi

Asynchronous Methods

Remote asynchronous user testing (AU)• Based on the idea of users reporting the critical incidents they experience

(Castillo, Hartson and Hix, 1998; Hartson and Castillo, 1998; Hartson, Castillo, Kelso and Neale, 1996)

• The users solve the tasks with the system• Simultaneously fill in a web-based form each time they encounter a usability problem• The form briefly explains what a usability problem is and how it can be categorized• In the form, they describe each problem and categorize it’s severity• Spatial and temporal separation

Remote asynchronous expert testing (AE)• Same method as AU but with users that have expertise in usability engineering (graduate

students in HCI)

Data: A set of described usability problems and a start and end time for each user

11

Institut for Datalogi 12

Survey of Remote Asynchronous Methods

Method Paper #

Auto logging 20, 28, 29, 30, 34, 35, 36, 37

Interview 10, 26, 29, 30, 35

Questionnaires 10, 19, 25, 28, 29, 30, 32, 34, 35, 36, 37

User-reported critical incident 1, 5, 6, 7, 13, 14, 31

Unstructured problem reporting 10, 19, 38

Forum 21

Diary 31

Institut for Datalogi 13

• Forhindringer i praksis

• Ideen om remote usability testing

• Realisering af ideen

Oversigt

Institut for Datalogi

• Forhindringer og udfordringer• Tekniske: Kan man få det til at fungere• Metodiske: giver det brugbare resultater• Praktiske: kan man gøre det i praksis

• Det skal vi arbejde med resten af dagen …

14

Realisering af ideen

Institut for Datalogi

aa

15