quill: A UI Design Case Study

A. Chris LongPostdoctoral Research Fellow

HCI Institute

2

Administrivia

Turn in Homework #1 Homework #2 assignment on course website

3

Outline

Gestures quill Design quill Architecture & Implementation Object-Oriented Design Long-running Tasks and Interfaces Recognition-based UIs How Gesture Recognition Works More quill Implementation

4

Gestures

Mark that invokes a command

5

Mark that invokes a command

Gestures

delete me

6

Mark that invokes a command

Benefits* Fast* Save screen space* Avoid “aerobic interface”* Specify operation and operand(s)

simultaneously

Gestures

delete me

7

Problems With Gestures

For users* Computer often misrecognizes

gestures* Difficult to learn and remember

gestures For designers

* Gestures are hard to design

8

Design

PrototypeTest

quill’s Place in the World

Who: Designer of pen-based UI

When: During design phase

What: Designing gestures

Why: To design better gestures

9

Another View of quill

Pen Input Display

OS

Application “back end”

User Interface

10

Designing quill

Design low-fi prototype Pilot test low-fi Redesign Implement Java version

11

Low-fi Prototype

12

Current Version

Tree view

Expanded view

Suggestion for improvement

13

Recognition Problem

14

Gestures, Packages, and Groups,

Oh My!

cut

Edit

MyApplication

View

copy …

…

Gesture

Gesturecategory

Gesturegroup

Gesturepackage

…

15

quill Architecture

JDK Core Gesture Objects

* Gesture* Gesture Category* …

Recognizer Gesture Analyses

Widgets

16

quill Architecture

Widgets

Gesture Analyses

Gesture & Recognition

Objects

Swing

Core JDK

17 Java/Swing

Implementation Overview

93 classes, 26,000 lines

Notice

Desktop

Main window

Tree display

management

windows

Clipboard

18

Object Oriented Design

Gesture Objects* Gesture* GestureCategory* GestureGroup* GesturePackage

Gesture Display Widgets* GestureDisplay* GestureInteractor* GestureCategoryDisplay* GestureGroupDisplay* GesturePackageDisplay* GestureCategoryThumbn

ailDisplay

19

OO Design

GestureObject

DefaultGestureObject

GestureContainer

AbstractGestureContainer

GestureCategory GestureGroup

GesturePackage

20

Analyzing Gestures

Problem: Analysis may take a long time Solutions?

21

Analyzing Gestures

Problem: Analysis may take a long time Strategies for analyzing gestures

* Run analysis in foreground, disable all actions

* Analyze in background- Disable all conflicting actions - Allow anything, cancel analyses- Allow anything

22

Multithreading

Benefits* ?

Drawbacks* ?

23

Multithreading

Benefits* Computation in

background- Can do very complex

tasks

* User can perform any action, any time

Drawbacks* Must synchronize data

across threads* Deadlock* Race conditions

24

Multiple Threads in Java

Thread class* Start separate thread that runs any method

synchronized keyword* Prevents a method from being called or

object from being accessed by more than one thread at once

SwingWorker class* Easy to run background thread that

produces a result

25

Recognition-Based UIs

Examples* Pen gesture* Handwriting* Speech

Advantages* ?

Disadvantages* ?

26

Recognition-Based UIs

Advantages* Natural input* Informal* Multiple interpretations

Disadvantages* No clear separation of data and commands* Ambiguous commands* Individual differences* Multiple interpretations

27

Dealing with Recognition

Confirm dangerous operations Multiple modes may help disambiguate

* Speech + pen, for example Provide undo/redo

28

How Does Recognition Work?

Training* Measure geometric

features of gestures* Compute average

feature values for each type of gesture

* Compute how important each feature is

Recognition (a.k.a. Classification)* Measure geometric

features of unknown gesture

* Compare with features of known gestures

* Pick closest known gesture

29

Rubine’s Feature Set

Initial angle Length of bounding

box diagonal Angle of bounding

box diagonal Distance between

first and last points Angle between first

and last points Length Total angle traversed

30

Rubine’s Feature Set (cont.)

Sum of absolute value of angle at each point

Sum of squared angle at each point (sharpness)

Square of maximum speed

Duration

31

Weakness of Feature-based Recognition

Feature set has to be right

32

Feature-based Training

Mean feature vector

Per-class covariance matrix

FifE

cE

eeic

cic

01 1

0ˆ

ˆˆ

ˆ

f

))(( ˆˆ

1

0ˆˆˆ

ˆ

jcejc

E

eiceicijc ff

c

ff

33

Training (cont.)

Average covariance matrix

Weights estimation

FjwF

iicijjc

1)(1

ˆ1

ˆ f

1

0ˆ

1

0 ˆ

ˆ

1C

cc

C

c c

ijc

ij

EC

E

34

Feature-based Classification

For all gesture classes c, compute

Return class with max vc

CcfwwvF

iiiccc

01

ˆ0ˆˆ

35

quill Implementation Issues (cont’d)

Suggestions (cont’d)* When do they disappear?

- User says to ignore- No longer apply

* When are two notices the same?- At time A, gestures X and Y are too similar- At time B, something happens and they aren’t

similar- At time C, they are similar again- Display notice again?

36

Implementation Issues (cont’d)

Annotations* Special purpose

vs. generic* How to compose?

Gesture display* Desktop windows

- Tiled- Standard MDI

* Display factory

37

Implementation issues (cont’d)

Generic propertiesvoid GestureObject.setProperty(String name, Object value)

Object GestureObject.getProperty(String name)

* What happens to properties like “isMisrecognized” when copy & paste?

* Persistent vs. transient

38

Implementation issues (cont’d)

Generic propertiesvoid GestureObject.setProperty(String name, Object value)

Object GestureObject.getProperty(String name)

* What happens to properties like “isMisrecognized” when copy & paste?

* Persistent vs. transient

39

Implementation Issues (cont’d)

Menus: enable/disable items, and dispatching commands* Context-dependent

menu items* Context-dependent

gesture drawing* Solution: “Chain of

command” pattern that follows selection

40

Questions?

Other topics* Grad school* Berkeley (city or university)* Speech UIs* More on pen-based UIs

The End

Design Patterns: Elements of Reusable Object-Oriented Software

by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides

http://www.cs.cmu.edu/~chrisl/http://guir.berkeley.edu/projects/quill/

Extras

43

Criticisms

Recognition is due to poor recognizers, which will eventually be perfect* Not for a long time, if ever* Learning and memory predictions still useful

Recognition will never be good enough* Already good enough for many users* Is improving over time

Typing is faster and more accurate than writing* Pens won’t replace keyboards, but are better in

many situations

44

Criticisms (cont.)

Gestures will always be too hard to learn and remember, and will never be useful enough* Gestures very often used on paper in many domains

Design tool doesn’t guarantee good gestures* Will provide useful advice* Will shorten design cycle

There are only a small number of common gestures* Application/domain-specific gestures* Personal shorthand

45

Criticisms (cont.)

Marking menus are better than free-form gestures* Can’t specify additional information with marking-

menus* Shapes are arbitrary* All straight lines and 45/90º angles

46

Criticisms (cont.)

What’s wrong with traditional GUI interaction techniques?* Pen mouse* Pen

- Finer control- Has other dimensions (e.g., pressure, tilt)

* Mouse- More buttons (sometimes)

47

Pen vs. Mouse

Kato, et al* Pen significantly faster & more accurate than mouse

for precise dragging* Pen significantly faster for pointing* Pen has directional dependencies; mouse doesn’t

Mackenzie, et al* Pen slightly faster for pointing* Mouse slightly faster for dragging