1

Computer-Aided Techniques for Extracting Usability Information from

User Interface Events

David M. Hilbert

Information and Computer ScienceUniversity of California, IrvineIrvine, California 92697-3425

dhilbert@ics.uci.eduhttp://www.ics.uci.edu/~dhilbert/

2

Overview

• Introduction

• Key Insights

• Discussion

• Conclusions

3

Introduction

Introduction

– Survey Topic

– Survey Context

– Survey Goals & Research Method

– Survey Results

• Key Insights

• Discussion

• Conclusions

4

Survey Topic

• UI events are a potentially fruitful source of usability data

– natural products of most modern graphical user interface systems

– easily and unobtrusively captured

– indicate user actions with respect to user interface components

– provide detailed information regarding timing and sequence of actions

• However, extracting useful information from UI events is difficult

– volume and detail can be overwhelming

– need to map lower level events into higher level events of interest

– need to map results of analysis back to features of the interface/application

• Need computer-aided support

5

Survey Context (Types of Evaluation)

User behavior/performance

User needs, desires, thought processes, & experiences

Usability metrics

Comparing design alternatives

Certifying conformance w/ standards

Predictive Evaluation

Observational Evaluation

Participative Evaluation

x

x

X

Reason for Evaluating

X

x

XX

x

X

XX

6

Survey Context (Data Collection)

On-line behavior:task times, % tasks completed, error rates, on-line help use, functions used

Non-verbal off-line behavior:eye movements, facial gestures, off-line document use & problem solving

Cognition/Understandingverbal protocols, comprehension-question answers, sorting task scores

Attitude/Opinion:post-hoc comments, questionnaire and interview results

Stress/anxiety:galvanic skin response (GSR), heart rate (ECG), event-related brain potentials (ERPs), EEG’s, etc.

Audio& Video

Recording

UI EventRecording

xX

Xx

X

X

Psycho-Physical

RecordingUsability Indicators

7

Survey Goals & Research Method

• Survey goals

– framework to help HCI practitioners/researchers compare and evaluate

– allow comparison of “classes” of systems, not just “instances”

• Research method

– search of academic and professional computing forums, including magazines, conference proceedings, journals

– initial set of papers used to identify “key” characteristics

– matrix constructed w/ techniques on one axis and attributes on the other

– visually detectable “clusters” of features suggested an initial classification

– attributes and classification iteratively refined based on further exploration

8

Survey Results

• Synch and Search

• Transformation

– Filtering

– Abstraction

– Recoding

• Analysis

– Counts and Summary Statistics

– Sequence Detection

– Sequence Comparison

– Sequence Characterization

• Visualization

• Integrated Support

9

Recoding

Synch & Search

Playback; Microsoft, SunSoft, & Apple Labs; DRUM; MacSHAPA; I-Observe

Counts & Summary Stats

30%15/hr.25 sec.}

MIKE UIMS; KRI/AG; MacSHAPA; Long-Term Monitoring; AUS

Abstraction

CHIME; Hawk; MacSHAPA; Remote CI’s; EDEM;

Filtering

Chen; CHIME; Hawk; Mac- SHAPA; Remote CI’s; EDEM;

10

Sequence Detection

TOP/G; LSA; Fisher’s Cycles; MRP; Chunk Detection; Expectation Agents; EDEM;

Sequence Characterization

Markov-based; Grammar-based; “Process Discovery”

Integrated SupportCounts & StatsFilteringAbstractionRecodingDetectionComparisonCharacterizationVisualization DRUM; Hawk; MacSHAPA

VisualizationCounts & StatsFilteringAbstractionRecodingDetectionComparisonCharacterization

I-Observe; AUS; UsAGE; MacSHAPA

Sequence Comparison

ADAM; UsAGE; MacSHAPA EMA; “Process Validation”

11

Key Insights

• Introduction

Key Insights

– Transformation is Critical (But Hard)

– The Grammatical Structure of Events

– Events at Multiple Levels

– Scattered and Missing Context

• Discussion

• Conclusions

• Related Work

12

Transformation is Critical (But Hard)

• Meaningful transformation is a prerequisite to meaningful analysis

– allows events and patterns of interest to emerge from the “noise”

– can significantly impact results of human and automated analyses

– can be helpful in linking results back to interface/application features

• However, meaningful transformation is hard to do

– the grammatical structure of UI events

– events occur at multiple levels of abstraction

– problems of scattered and missing contextual information

13

The Grammatical Structure of Events

• A grammar for activating a print job

Print -> PrintToolbarButtonPressed OR (PrintDialogActivated THEN PrintDialogOkayed)

PrintDialogActivated -> PrintMenuItemSelected OR PrintAcceleratorKeyEntered

• Assume the lexical elements in this language are

A: indicating "print toolbar button pressed”B: indicating "print menu item selected”C: indicating "print accelerator key entered”D: indicating "print dialog okayed"

• What do these “sentences” indicate?

AAAA CDAAA ABDBDA BDCDACD CDBDCDBD

14

The Grammatical Structure (cont’d)

• These sequences are semantically equivalent

AAAA CDAAA ABDBDA BDCDACD CDBDCDBD

– all occurrences of ‘A’ indicate an immediate print job activation

– all occurrences of ‘BD’ or ‘CD’ indicate a print job activated by using the print dialog and then okaying it

• Notice, however

– each sequence contains a different number of lexical elements

– some have absolutely no lexical elements in common

– the lexical elements occupying the first and last positions differ

• Significance:

– most techniques do not address grammatical issues, and thus, are overly sensitive to such lexical differences

15

Goal/Problem-Related(e.g. placing an order)

Domain/Task-Related(e.g. providing address information)

Abstract Interaction Level(e.g. providing input values)

Window System Events(e.g. key events, mouse events, input focus events)

Input Device Events(e.g. hardware-generated key or mouse interrupts)

Physical Events(e.g. fingers presses, hand movements)

Events at Multiple Levels

16

Events at Multiple Levels (cont’d)

• Higher level events do not (in themselves) reveal their composition from lower level events

– recording only higher level events results in information loss

• Lower level events do not (in themselves) reveal how they combine to form higher level events

– either higher level events must be recorded in conjunction with lower level events, or

– there must be some model (e.g. a grammar) to describe how lower level events combine to form higher level events

• Significance:

– most techniques do not address issues of events at multiple levels, and thus, mapping from lower level events to higher level events can be difficult

17

Scattered and Missing Context

• Given a transcript of a conversation between A & B at a car show, identify A’s favorite cars based on utterances in the transcript:

– Example 1: "The Lamborghini is one of my favorite cars".

• everything you need to know is contained in a single utterance

– Example 2: "The Lamborghini".

• need access to prior context — probably a response to a prior utterance, e.g., ”Which car here is your least favorite?"

– Example 3: "That is one of my favorite cars".

• need ability to de-reference an indexical — information carried by the indexical “that” may not be available in any of the utterances in the transcript, but was available at the time of the utterance

– Example 4: "That is another one".

• need access to prior context and ability to de-reference an indexical

18

Scattered and Missing Context (cont’d)

• Analogous UI event examples:

– Example 1’: PrintToolbarButtonPressed

• this event carries enough information to indicate the action taken

– Example 2’: CancelButtonPressed

• what was canceled? indicates a response to a prior event, e.g., a prior PrintMenuItemSelected event.

– Example 3’: ErrorDialogActivated

• what’s the error? query the dialog for its error message; this is similar to de-referencing an indexical if you think of the error dialog as “pointing at” an error message that doesn’t appear in the event stream

– Example 4’: ErrorDialogActivated

• assuming the error message is “Invalid Command”, then the information needed to interpret the significance of this event is not only found by de-referencing the indexical, but must be supplemented by information available in prior events

19

Scattered and Missing Context (cont’d)

• Significance:

– context is critical in interpreting the significance of UI events

• sometimes context is spread across multiple events

• sometimes context is missing altogether from the event stream

• however, context is usually available “for the asking” from the user interface system at the time of the events, but not afterwards

– most techniques do not perform transformation and analysis in-context, and thus, cannot access contextual information critical in interpretation

20

Discussion

• Introduction

• Insights

Discussion

– Problems

– Non-Answers

– Requirements for a Successful Solution

• Conclusions

• Related Work

21

Problems

• Meaningful transformation appears to be a prerequisite to meaningful analysis

• However, of the 25+ techniques surveyed, only a handful provide any assistance in automating transformation

– e.g., CHIME, Hawk, & EDEM

• Unfortunately, all suffer from some combination of the following problems

– failure to perform transformation in-context

– failure to allow analysis at multiple levels of abstraction

– lack of assistance in mapping results of analysis back to features of the interface/application being evaluated

– lack of support for end-user modifiability and reuse of transformation and analysis mechanisms

22

Non-Answers

• Why not use a UIMS? (e.g. MIKE, KRI/AG, UsAGE)

– UIMS’s model relationships between UI actions and application features

– however, more general techniques are required

• Why not rely on hand-instrumentation? (e.g. LTM, EMA)

– hand-instrumentation allows higher level and application-specific events to be reported, and is more general than a UIMS-based approach

– however, too much burden placed on application developers, and

– much useful usability-related information is not available to applications

• e.g., shifts in input focus, mouse movements, specific UI actions used to invoke application features, etc.

• Why not just rely on users? (e.g. User-Identified CI’s)

– users often fail to recognize when their behavior violates expectations about proper usage (Smilowitz et al., 1994)

23

Requirements for a Successful Solution

• Need an approach that:

– does not require developers to adopt a particular UIMS

– does not require developers to call an API to report every potentially interesting event

– does not rely (solely) on users to perform event filtering and abstraction

• The approach must also address:

– transformation in-context

– analysis at multiple levels of abstraction

– mapping of results back to features of the interface/application

– end-user modifiability and reuse of transformation and analysis mechanisms

24

Conclusions

• Introduction

• Insights

• Discussion

Conclusions

– More Research

– A Hypothesis

– Open Issues

25

More Research

• If meaningful usability information is to be extracted automatically from UI events, then more research is needed to address:

– how to allow transformation and analysis to be performed in-context without significantly impacting application performance, evolution, and deployment.

– how to establish a reasonable mapping between lower level events and higher level events of interest that is maintainable as the application interface and functionality evolves.

– how to establish a reasonable mapping between events and application features that is also maintainable as the application interface and application functionality evolves.

– how to make transformation and analysis mechanisms easily adaptable and reusable by investigators wishing to employ them.

26

A Hypothesis

• Performing transformation and analysis automatically and in-context has the potential of greatly reducing data collection

• As a result, such an approach might scale up to large-scale and ongoing use over the Internet

– lift current restrictions on evaluation size, location, duration

– help developers and managers answer such empirical questions as:

• how thoroughly has beta testing exercised relevant features?

• how are application features actually being used?

– which features warrant more/less development and testing effort?

– which features if modified/added/deleted are most likely to impact application utility, usability, and productivity?

• does actual usage match expectations?

– how can the design be improved to better match actual usage?

27

Open Issues

• Analysis might address single or multiple users at one time

– The techniques surveyed here primarily address single users

– If events reflect users associated with them, these techniques might be extended to address multiple users

• Analysis might address synchronic and diachronic aspects of processes

– Characterizing a user’s behavior in a single session in terms of a “process” addresses synchronic aspects of processes.

– Characterizing changes in a user’s process as it evolves over time addresses diachronic aspects of processes.

– The techniques surveyed here primarily address synchronic aspects of processes.

28

Related Work

29

Related

• Model-based debugging/testing (e.g. EBBA & TSL)

• Beta test data collection (e.g. Aqueduct Profiler)

• API usage monitoring (e.g. HP/Tivoli ARM API)

• Event histories and undo (e.g. Kosbie & Myers, 1994)

• Layered interaction models (e.g. Nielsen, 1986, Taylor, 1988)

• Command & task-action grammars (e.g. Moran, 1981, Payne & Green, 1986)

• UI testing/debugging (e.g. Mercury Interactive X/WinRunner, SunTest JavaStar)

• Process validation techniques (Cook & Wolf, 1997)

• Process discovery techniques (Cook & Wolf, 1996)

30

Somewhat Related

• Collaborative remote usability (e.g. “Remote Evaluation” web site)

• Simple event capture (e.g. Windows SPY)

• Macro recording/playback (e.g. Windows Macro Recorder)

• Programming by demonstration (e.g. Cypher et al., 1994)

• Task recognition and assisted completion (e.g. Cypher, 1991)

• Enterprise management (e.g. TIBCO Hawk)

• Product condition monitoring

• Firewall monitoring, intrusion detection

• Network monitoring/diagnostics

• Trade monitoring

31

Supporting Technologies

• Transport mechanisms

– Hypertext transfer protocol (HTTP)

– Mobile agent infrastructures (e.g. ObjectSpace Voyager)

• Event notification

– Simple mail transfer protocol (SMTP)

– More advanced event notification systems (e.g. TIB/Rendezvous)

32

Background & Definitions

33

“Usability”

• “Usability” has multiple components and is traditionally associated these five usability attributes (Nielsen, 1993):

– Learnability: can users rapidly figure out how to accomplish tasks?

– Efficiency: once learned, can tasks be accomplished quickly?

– Memorability: can the casual user return after some period of time without having to relearn everything?

– Errors: are error rates reasonably low? can users easily recover from errors? are catastrophic errors prevented?

– Satisfaction: are users subjectively satisfied when using the system? Do they like it?

34

Usability Evaluation & Usability Data

• Usability evaluation:

– the act of measuring — or detecting potential “issues” affecting — usability attributes of a system or device with respect to particular users, performing particular tasks, in particular contexts

– values of usability attributes will typically vary depending on the background knowledge and experience of users, tasks for which the system is used, and context in which it is used

• Usability data:

– any information useful in measuring — or detecting potential “issues” affecting — usability attributes of a system being evaluated

35

Usefulness

• Usefulness = Utility + Usability (Grudin, 1992)

– “utility” is (roughly) the question of whether system functionality can in principle support the needs/desires of users

– “usability” is (roughly) the question of how satisfactorily users can make use of system functionality

• Usability evaluations often uncover issues relating to both utility and usability, and thus more properly address “usefulness”

36

Evaluation Approaches

• Predictive evaluation

– predict values of usability attributes based on psychologically-inspired models (e.g. GOMS or the Cognitive Walkthrough), or based on formal reviews by experts

– key strength is the ability to produce results with non-functioning design artifacts and without actual users (not so great for evaluating “utility”)

• Observational evaluation

– measure usability attributes by observing users interacting with prototypes or fully functioning systems

– key strength is the ability to uncover aspects of user behavior and performance difficult to capture using other techniques

• Participative evaluation

– collect information regarding usability attributes directly from users based on their subjective reports

– key benefit is the ability to capture aspects of users’ needs, desires, thought processes, and experiences difficult to obtain otherwise

37

A Spectrum of HCI Events

38

Comparison of Techniques

39

Synch and Search

• Strengths

– combines advantages of detailed on-line behavior data with more semantically rich data

– allows searches in one source to locate supplementary data in others

– allows context missing from events to be used in interpretation

– facilitates communication of results back to developers

• Limitations

– requires video recording equipment, operator, observer

– equipment can be expensive to obtain/operate

– may make subjects self-conscious and affect behavior

– often produces massive amounts of data that is time-consuming to analyze

– places limitations on evaluation context, size, duration, etc.

40

Transformation

• Strengths

– allows events and patterns of interest to emerge from the “noise”

– can significantly impact results of human and automatic analysis

– if performed “in-context”, could support large-scale remote collection

• Limitations

– if performed “in-context”, important information may be lost

– approaches relying on users are even more likely to drop important data

– most techniques don’t support transformation “in-context”

– meaningful interpretation, particularly without the help of users, is challenging

41

Counts and Summary Statistics

• Strengths

– with the large number of potential counts and summary statistics, built-in support makes some sense

• Limitations

– most approaches do not allow users to modify or define their own counts, summary statistics, and reports

– the MIKE UIMS approach relies on information about the mapping between application commands and UI actions not typically available

– the MacSHAPA approach does not operate in-context and thus misses potentially important contextual information

– the AUS approach does not make use of application-specific information and thus is limited in what it can say with regard to application features

42

Sequence Detection

• Strengths

– addresses sequential information in addition to isolated events

– ESDA techniques may be helpful in “discovering” unanticipated patterns

• Limitations

– ESDA techniques produce large amounts of output that is difficult and time-consuming to interpret

– the efficacy of ESDA techniques at uncovering usability issues has been called into doubt (Coumo, 1994)

– non-ESDA techniques require more effort to specify sequences of interest

43

Sequence Comparison

• Strengths

– allows actual usage to be compared against some model or trace of “ideal” or expected usage to identify potential usability issues

– particularly attractive when “ideal” usage can be specified “by demonstration”

• Limitations

– some approaches assume short, pre-segmented sequences (e.g. ADAM)

– some approaches assume a reasonable correspondence will actually exist between separate traces (e.g. UsAGE).

– some approaches assume a “complete” model of all possible/expected sequences (e.g. EMA)

– all approaches assume reasonably abstract events and thus presuppose transformation or careful hand instrumentation.

– all approaches require expert user interpretation of output to determine whether sequences are interestingly similar or different (especially ADAM)

44

Sequence Characterization

• Strengths

– provides insight into the sequential structure of UI event sequences

– Markov-based techniques provide insight into transition probabilities between particular features or sets of features

– grammar-based techniques are powerful for performing event abstraction

• Limitations

– typically requires extensive human involvement, particularly grammar-based techniques

– automated approaches (e.g. “process discovery”, Cook and Wolf, 1996) appear to be less able to produce models meaningful to investigators (Olson et al., 1994), at least without careful event stream transformation ahead of time

– investigators more likely to want to see actual instances of subsequences in order to count and analyze them rather than to see a single abstract model (such as a grammar of FSM) to summarizes them all

45

Visualization

• Strengths

– allows humans to use innate visual analysis capabilities

– particularly useful in linking results of analysis back to features of the interface

• Limitations

– most non-standard graph visualizations must be created by hand

46

Integrated Support

• Strengths

– reduces the burden of managing multiple artifacts and media types and the creation and composition of various transformation and analysis techniques

• Limitations

– existing environments do not address event capture, so contextual information critical in proper transformation and analysis is missing

– most environments don’t allow for flexible definition and reuse of automated transformations and analyses (except for Hawk)

47

Examples of Techniques

48

Synch and Search

• Playback (Neal & Simons, 1983)

– events, coded observations, play back

• DRUM (Macleod et al., 1993)

– events, coded observations, video

• Usability labs at Microsoft, Apple, SunSoft (Weiler et al., 1993)

– events, coded observations, video, “highlight videos”

• I-Observe (Badre et al., 1995)

– events, video, regular expression search, visualizations

49

Transformation

• “Intrinsic” monitoring (Chen, 1990)

– automatic & investigator-specifiable filtering, limited widget context

• User-identified “Critical Incidents” (Hartson et al., 1996)

– only events surrounding user-identified CI’s reported, users provide context

• CHIME (Badre & Santos, 1991)

– investigator-specifiable filtering, limited abstraction via notion of IU’s

• Hawk (Guzdial, 1993)

– AWK used to filter and abstract log files, no context

• MacSHAPA (Sanderson & Fisher, 1994)

– database query and manipulation language, manual techniques, no context

• EDEM (Hilbert & Redmiles, 1998)

– investigator-specifiable filtering & abstraction, CI filtering, grammar-based abstraction, access to context via standard software component model

50

Counts and Summary Statistics

• MIKE UIMS (Olsen & Halverson, 1988)

– computation and reports of task performance time, mouse travel, command frequency, command-pair frequency, cancel and undo, physical device swapping, and more

• MacSHAPA (Sanderson & Fisher, 1994)

– support for user-specifiable counts, summary statistics, and reports

• Automatic Usability Software (AUS) (Chang & Dillon, 1997)

– use of help, cancel and undo, mouse travel, mouse click density per window, and more

51

Sequence Detection

• Task-oriented parser/generator (TOP/G) (Hoppe, 1988)

– grammar-based parser/generator based on TAG (Payne & Green, 1983)

• Exploratory sequential data analysis (ESDA) techniques (Sanderson & Fisher, 1994)

– techniques sensitive to sequentially separated patterns

• e.g. Fisher’s cycles (Fisher, 1991), lag sequential analysis (LSA) (Sackett, 1978)

– techniques sensitive to strict transitions between events

• e.g. Markov-based techniques, maximal repeating patterns (MRP), (Siochi & Hix, 1991), log linear analysis (Gottman & Roy, 1990), time-series analysis (Box & Jenkins, 1976)

• Automatic chunk detection (Santos et al., 1994)

• Expectation agents (Girgensohn et al., 1994)

• EDEM (Hilbert & Redmiles, 1998)

52

ESDA Examples

• Fisher's Cycles

– Source sequence: ABACDACDBADBCACCCD

– Begin event: A

– End event: D

• Output:

ABACDACDBADBCACCCD

ABACDACDBADBCACCCD

ABACDACDBADBCACCCD

ABACDACDBADBCACCCD

• Notice:

– no occurrence of B in A->D cycles

– grammatical recoding of repeated C’s makes last cycle identical to first two

– helps investigators locate similar patterns in “noisy” data

53

• Lag Sequential Analysis (LSA)

– Source sequence: ABACDACDBADBCACCCD

– Key event: A

– Target event: D

– Lag(s): -4 through +4

• Output:

• Notice:

– count of 2 at Lag = +2 corresponds to the ACD cycles in Fischer's cycles

– assuming a similar recoding operation, this count would increase to 3

– helps identify correlations between events (that may be causally related) missed by other techniques more sensitive to strict transitions

ESDA Examples (cont’d)

54

ESDA Examples (cont’d)

• Maximal Repeating Patterns (MRP):

– Source Sequence: ABACDACDBADBCACCCD

• Output:

• Notice:

– MRP is similar in spirit to Fisher’s cycles and LSA, however, the investigator does not specify particular events of interest

– ACCCD subsequence identified in previous examples not identified by MRP

55

Sequence Comparison

• ADAM (Finlay & Harrison, 1990)

– associative memory, trained on example sequences, provides classification of input sequences and confidence/distance measures

• MacSHAPA (Sanderson & Fisher, 1994)

– allows sequences to be “optimally” aligned in spreadsheet cells, automatic and manual techniques

• UsAGE (Ueling & Wolf, 1995)

– graphical alignment of event traces of expert and non-expert users to compare behavior

• EMA (Balbo, 1996)

– unexpected user behavior identified using an investigator-specified grammar-based model of all possible (or expected) paths through a UI

56

Sequence Characterization

• A Markov-based technique (Guzdial, 1994)

– sets of commands mapped to “stages” of application use

– transition matrix constructed to represent transition probabilities

– process model constructed with probabilities assigned to transitions

– steady state vector represents relative “command set” usage

– comparison w/ expected usage matrix yields “command preference”

• A statistical and grammar-based technique (Olson et al., 1993)

– log linear and lag sequential analysis used to detect patterns of interest

– patterns suggest definite clause grammar rules

– grammar used to rewrite (i.e. abstract & recode) subsequences

– technique applied iteratively

– resulting grammar characterizes sequential structure of events

57

Visualization

• Event sequence-based

– e.g. timelines, event alignment visualizations

• Static 2D and 3D graphs

– e.g., keystrokes per window, mouse clicks per window (Chang & Dillon, 1997), relative command frequencies (Kay & Thomas 1995), relative command frequencies over time (Kay & Thomas 1995)

• Static 2D effects superimposed on abstract coordinate spaces representing the interface:

– e.g. mouse travel patterns (Buxton et al., 1983, Chang & Dillon, 1997), mouse click locations (Chang & Dillon, 1997)

• Static and dynamic 2D and 3D effects superimposed on visual depictions of the interface (Guzdial, et al., 1994):

– e.g. static highlighting indicating mouse click locations, dynamic highlighting indicating mouse click activity over time, 3D representation of mouse click location and “density”

58

Meeting.db

Amplify

Resolve Issue

Identify Problem

Digress

Recapitulate

Amplify

Identify Issue

Amplify

Resolve

Identify

Amplify

Digress

Amplify

Amplify

Amplify

Amplify

Resolve

Identify Problem

Identify

Identify

Identify

Identify Issue

Digress

Recapitulate

Recapit

Resolve Issue

Resolve

Resolve

Resolve

MeetingA MeetingB

GotFocus(Name)

Key(Name, ‘D’)

Key(Name, ‘a’)

GotEdit(Street)

ValueProvided(Name, “David”)

LostEdit(Name)

GotEdit(Name)

UI Events Abs. Interaction Events Task-Related. Events

Key(Name, ‘v’)

Key(Name, ‘i’)

Key(Name, ‘d’)

LostFocus(Name)

GotFocus(Street)

Key(Street, ‘1’)

Key(Street, ‘3’)

Key(Street, ‘8’)

AddressSectionStarted()

NameProvided(“David”)

GotEdit(Quantity)

ValueProvided(ZIP, “90740”)

LostEdit(ZIP)

LostFocus(ZIP)

GotFocus(Quantity)

Key(Quantity, ‘1’)ZIPProvided(“90740”)

AddressSectionCompleted()

OrderForm.db

Visualization Examples

Edit Menu Use

Font Menu Use

00:01:00 00:02:00 00:03:00 00:04:00 00:05:00

Timeline: WordProcessor.db

MeetingA MeetingB

Alignment: Meeting.db

Amplify

Amplify

Amplify

Amplify

Identify Problem

Identify Issue

Identify Issue

Identify Issue

Digress

Digress

Recapitulate

Resolve Issue

Resolve Issue

Identify Problem

Digress

Recapitulate

Amplify

Identify Issue

Amplify

Resolve Issue

Identify Issue

Amplify

Digress

Resolve Issue

Amplify

59

Visualization Examples (cont’d)

60

Integrated Support

• MacSHAPA (Sanderson et al. 1994)

– data import and export

– event/observation/video synch and search

– user-specifiable filtering, abstraction, and recoding

– user-specifiable counts and summary statistics

– sequence detection, comparison, and characterization

– general purpose database query and manipulation language

– visualizations and reports

• DRUM (Macleod et al. 1993)

– event/observation/video synch and search

– pre-defined counts and summary statistics

– domain-specific data management of evaluation artifacts.

61

Integrated Support (cont’d)

• Hawk (Guzdial 1993)

– creation, debugging, and execution of reusable, automated filtering, abstraction, and recoding scripts

– integrated management and display of event logs and analysis scripts

62

Summary of Surveyed Techniques

Synch & Search:Synchronize and cross-index UI events w/ other data sources, e.g. video & coded observations, allowing information in one source to supplement information in others

Playback (Neal & Simmons, 1983)DRUM (Macleod et al., 1993) Microsoft Lab (Hoiem & Sullivan, 1994)Apple Lab (Weiler, 1993)SunSoft Lab (Weiler, 1993)I-Observe (Badre et al., 1995)

Transformation:Transform event streams to facilitate human and/or automatic analysis

•filtering: subtraction of information•abstraction: synthesis of information•recoding: creating a new event stream

Intrinsic Monitoring (Chen, 1990)CHIME (Badre & Santos, 1991)Hawk (Guzdial, 1993)MacSHAPA (Sanderson & Fisher, 1994)Remote “CI”s (Hartson et al., 1996)EDEM (Hilbert & Redmiles, 1997)

Counts & Summary Stats:Compute and report summaries of usability-related “metrics” such as feature usage, error frequencies, use of help system, etc.

MIKE UIMS (Olsen & Halversen, 1988)MacSHAPA (Sanderson & Fisher, 1994)AUS (Chang & Dillon, 1997)

Aqueduct Profiler (Aqueduct Software, 1997)ARM API (HP/Tivoli, 1997)

63

Summary (cont’d)

TOP/G (Hoppe, 1988)LSA (Sackett, 1978)Fisher's Cycles (Fisher, 1988)MRP (Siochi & Hix, 1991)Log linear analysis (Gottman & Roy, 1990)Time-series analysis (Box & Jenkins, 1976)Chunk detection (Santos et al., 1994)Expectation Agents (Girgensohn et al., 1994)EDEM (Hilbert & Redmiles, 1997)

TSL (Rosenblum, 1991)EBBA (Bates, 1995)GEM (Mansouri-Samani & Sloman, 1997) Amadeus (Selby et al., 1991)YEAST (Krishnamurthy & Rosenblum, 1995)

Sequence Detection:Identify occurrences of concrete or abstractly defined sequences within sequences of UI events that could indicate potential usability issues

Sequence Comparison:Compare actual sequences of UI events against models or traces of "ideal" sequences to identify potential usability issues

ADAM (Finlay & Harrison, 1990)MacSHAPA (Sanderson & Fisher, 1994) UsAGE (Ueling & Wolf, 1995)EMA (Balbo, 1996)

Process Validation (Cook & Wolf, 1997)

64

Summary (cont’d)

Markov-based (Guzdial, 1993)Grammer-based (Olson et al., 1994)

Process Discovery (Cook & Wolf, 1995)

Sequence Characterization:Construct an abstract model to characterize, or summarize, interesting sequential features of UI event sequences

Timelines (Sanderson & Fisher, 1994)Event alignment (Ueling & Wolf, 1995)Keystrokes/window (Chang & Dillon, 1997)Mouse clicks/window (Chang & Dillon, 1997)Command freq. (Kay & Thomas 1995)Command freq./time (Kay & Thomas 1995)Mouse travel (Buxton et al., 1983) Keystroke locations (Chang & Dillon, 1997)Mouse click locations (Chang & Dillon, 1997)Static & dynamic, 2D & 3D visualizationssuperimposed on UI (Guzdial et al, 1994)

Visualization:Present results in a such a way as to facilitate use of innate visual analysis capabilities in interpreting results; Link results back to the interface

MacSHAPA (Sanderson & Fisher, 1994)DRUM (Macleod et al., 1993)Hawk (Guzdial, 1993)

Integrated Support:Flexible composition of transformations, analyses, and visualizations; Domain-specific data management capabilities