gestures as point clouds: a $p recognizer for user ... · vatavu, anthony, & wobbrock. gestures...

Vatavu, Anthony, & Wobbrock. Gestures as Point Clouds: A

$P Recognizer for User Interface Prototypes, ICMI 2012

1

Gestures as Point Clouds:

A $P Recognizer for

User Interface Prototypes

Radu-Daniel Vatavu (University Stefan cel Mare of Suceava, [email protected])

Lisa Anthony (UMBC Information Systems, [email protected])

Jacob O. Wobbrock (Information School | DUB Group, University of Washington,

[email protected])

ICMI 2012



2

Pen and Finger Gesture Input

My Word Coach (DS)

Obenkyo (Android)

Mr. Spiff’s Revenge (PC)



3

$-Family of Gesture Recognizers

$-family of recognizers ($N, $P) are simple, accurate

multistroke recognizers (built on $1)

– $1 is a simple, accurate unistroke recognizer ^

– “one dollar” = cheap, fast, easy

– $N = multistrokes (e.g., n strokes) *

– $P = point clouds (e.g., p points)

^ Wobbrock, J.O., Wilson, A., and Li, Y. Gestures without Libraries, Toolkits, or Training: A $1 Recognizer for User Interface Prototypes. Proc. UIST’2007, p.159-168. * Anthony, L. and Wobbrock, J.O. (2010). A lightweight multistroke recognizer for user interface prototypes. Proceedings of Graphics Interface (GI '10). Ottawa, Ontario (May 31-June 2, 2010). Toronto, Ontario: Canadian Information Processing Society, pp. 245-252.

$1 $N $P



4

$-Family are Template Matchers

Drawn candidate. Resampled. Rotated to 0°. Scaled, translated.

Drawn template. Normalized. Alignment before GSS.Optimal alignment.

Candidate

Template

Template matcher: point per point correspondence



5

$N Multistroke Recognizer

Multi-stroke representation of an “x”



6

$N Multistroke Recognizer

Multi-stroke permutations of an “x”

$N’s internal representation of an ‘x’ the 8 ways to write an ‘x’ gesture



7

Gesture Corpus Tested

Mixed Multistroke Corpus

(MMG) ^

– 20 able-bodied adults

– Gesture and UI oriented

symbols (1-3 strokes)

$N 98% accurate with

8 training examples per

symbol

MMG Corpus

^ Anthony, L. and Wobbrock, J.O. 2012. $N-Protractor: A Fast and Accurate Multistroke Recognizer. Proceedings of Graphics Interface (GI’2012), Toronto, Canada, p.117-120.



8

Costly to Represent Permutations

$N experiences exponential growth in number of permutations to represent.

1

10

100

1000

10000

100000

1000000

10000000

100000000

1E+09

1E+10

1 2 3 4 5 6 7 8 9 10

Nu

mb

er

of

Pe

rmu

tati

on

s (l

oga

rith

mic

sca

le)

Number of Strokes in the Gesture



9

$P Point Cloud Recognizer

$P abstracts execution details: number of strokes, stroke order, and stroke direction.



10

Matching as an Assignment Problem

Hungarian algorithm * solves assignment problem

^ Papadimitriou, C. H., and Steiglitz, K. Combinatorial optimization: algorithms and complexity. Dover Publications, Mineola, New York, USA, 1998.



11

Evaluating the Hungarian Algorithm

Hungarian outperforms template-matching competitors on multistroke recognition for both user-dependent and user-independent cases.



12

Evaluating the Hungarian Algorithm

But, Hungarian is too costly in terms of execution time, and complexity.

Feasible labelings

Alternating trees Alternating

paths

Equality graphs



13

Approximation of Hungarian

Approximate ideal min-cost matching with greedy

techniques



14

Selecting the Best Approximation

Greedy5 with ε of 0.50 is the best performer, even outperforms Hungarian.



15

$P Matching Demo: Unistroke



16

$P Matching Demo: Multistroke



17

Performance of $P

– $P is 99% accurate with 5 training examples per symbol

(user-dependent)

– $N was 98% accurate with 8 training examples per symbol

– $P is also 99% accurate with 10 participants' data

(user-independent)



18

Performance of $P


(user-dependent)



(user-independent)

$P now state of the art $-family recognizer for multistrokes.



19

Performance of $P


(user-dependent)



(user-independent)




20

Performance of $P


(user-dependent)



(user-independent)




21

Limitations of $P

$P is not rotation-invariant

$P is not direction-invariant

21



22

$-Family is Open Source

Reference implementations available for $1, $N, $P

(pseudocode, JavaScript, C#)

Supports independent app development

– Rapid pick-up by community ($1, $N)

– AlphaCount iPhone app ($N)

– iOS port of the recognizer ($1, $N)

$P has ~70 lines of pseudo-code

– 50% reuse $1 pseudo-code

AlphaCount (iOS)



23

$-Family Cheat Sheet

Cheat Sheet for $-family helps app developers choose best approach for app context.



24

Summary

$-family of recognizers has a

new member: $P.

$P is highly accurate.

$P uses simple geometric principles

to match gestures.

$P is invariant to (1) number of strokes, (2) stroke type,

(3) stroke direction, and (4) stroke ordering.

$P saves both storage space and execution time for

multistroke recognition, compared to $N.

$-family of recognizers continues tradition of easy-to-

use, open-source, accessible gesture recognition.



25

Questions?

Contact:

– Radu-Daniel Vatavu, PhD

• [email protected]

– Lisa Anthony, PhD


– Jacob O. Wobbrock, PhD


Online demo:

– http://depts.washington.edu/aimgroup/proj/

dollar/pdollar.html



26

References

1. L. Anthony and J.O. Wobbrock. A lightweight multistroke recognizer for user interface

prototypes. Proceedings of Graphics Interface ’10 (Ottawa, Canada, May 31-June 2,

2010), 245-252. Canadian Information Processing Society, 2010.

2. Anthony, L. and Wobbrock, J.O. 2012. $N-Protractor: A Fast and Accurate Multistroke

Recognizer. Proceedings of Graphics Interface (GI’2012), Toronto, Canada, p.117-120.

3. Y. Li. Protractor: a fast and accurate gesture recognizer. Proceedings of ACM SIGCHI

Conference on Human Factors in Computing Systems ’10 (Atlanta, Georgia, April 10-

15, 2010), 2169-2172. ACM Press, 2010.

4. Papadimitriou, C. H., and Steiglitz, K. Combinatorial optimization: algorithms and

complexity. Dover Publications, Mineola, New York, USA, 1998.

5. J.O. Wobbrock, A.D. Wilson, and Y. Li. Gestures without libraries, toolkits or training: A

$1 recognizer for user interface prototypes. Proceedings of ACM Symposium on User

Interface Software and Technology ’07 (Newport, Rhode Island, October 7-10, 2007),

159-168. ACM Press, 2007.

gestures as point clouds: a $p recognizer for user ... · vatavu, anthony, & wobbrock. gestures...

Documents