pen-centric shorthand interfaces

1

Pen-CentricShorthand Interfaces

Charles C. TappertSeidenberg School of CSIS, Pace

University

2

Themes of Presentation Online Handwriting Recognition and Pen

Computing Tutorial Historical Research – undertaken for the

Palm-Xerox Patent Infringement Lawsuit Recent Research - Enhanced Pen-Centric

Shorthand Interfaces can have benefits DPS dissertation could extend M.S. thesis

3

Enhanced Pen-Centric Shorthand Interfaces

Can use word/phrase shorthand to speed text input

Can provide critical infrastructure for many pen-centric applications

Can enhance natural pen-centric interactions for many applications

Will have greatest impact on the utility of applications running on small mobile devices

4

Part 1: Online (Pen-Centric)Handwriting Recognition

Written Languages and Handwriting Properties The Fundamental Property of Writing Handwriting Recognition Difficulties Online (Pen-Centric) Handwriting Recognition Online more accurate than Offline Recognition Online Info Can Complicate Recognition

Process Design Tradeoffs / Design Decisions Computer Problems in English

5

Written Language and Handwriting Properties

Alphabet Letters, digits, punctuation, special

symbols Writing is a time sequence of strokes

Stroke – writing from pen down to pen up Usually complete one character before

beginning the next Spatial order – e.g., in English left to right

6

Fundamental Property of Writing

Differences between different characters are more significant than differences between different drawings of the same character

This makes handwritten communication possible

Can there be exceptions – say, different characters written identically?

7

Fundamental Property of Writing

in English Property holds within subalphabets of

uppercase, lowercase, and digits, but not across them

“I”, “l”, and “1” written with single vertical stroke

“O” and “0” written similarly with an oval

8

Handwriting Recognition Difficulties Shape, size, and slant variation Similarly shaped characters – U and V Careless writing

in the extreme, almost illegible writing Resolving difficult ambiguities

requires sophisticated recognition algorithms, syntax/semantics

9

Electronic tablets invented in late 1950s Digitizer and display in separate surfaces

Pen Computers arrived in 1980s Combined digitizer and display Brought input and output into one surface Immediate feedback via electronic ink Created paper-like interface

Online (Pen-Centric) Handwriting Recognition

10

Tablet Digitizers – Dynamic Information

Pen down – indication of inking X-Y coordinates as function of time

Sampling rate: 100 points per second Resolution: 200 points per inch

11

Early Pen-Centric Interface

Different surfaces for input and output

Rand system, about 1959

12

Pen Computers

IBM vision Paper-like interface,

1992

Microsoft Tablet PC Launched, 2001

13

Pen-Centric PDAs

Early Palm Pilot

Palm Tungsten T3 and Sony Clié TH55

14

Online (Pen-Centric) Handwriting Recognition

Machine recognizes the writing as the user writes

Digitizer equipment captures the dynamic information of the writing

Stroke number, order, direction, speed A stroke is the writing from pen down to pen up

15

Online (Pen-Centric) more accurate than Offline (Static)

Recognition Can use both dynamic and static

information Can often distinguish between similarly

shaped characters E.g., 5 versus S where the 5 is usually

written with two strokes and the S with one stroke

16

Online Information Can Complicate Recognition Process

Large number of possible variations E can be written with one, two, three, or four

strokes, and with various stroke orders and directions

A four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)

17

Online Information Can Complicate Recognition Process

Other variations

18

Online Information Can Complicate Recognition Process

Segmentation ambiguities character-within-character problem lowercase d might be recognized as a cl if

drawn with two strokes that are somewhat separated from one another

19

Design Tradeoffs/Decisions No constraints on the user

Machine recognizes user's normal writing User severely constrained

Must write in particular style such as handprint

Must write strokes in particular order, direction, and graphical specification

20

English Writing Styles Handprint

Uppercase – about 2 strokes per letter Lowercase- about 1 stroke per letter

Cursive Script Usually less than 1 stroke per letter Delayed crossing and dotting strokes

21

Computer Problems in English

Constrained Handprint Printing one symbol per box – form filling Printing on lines – symbols can touch or

overlap Unconstrained Handprint

No lines and symbols can touch or overlap

Cursive Script Mixed Printing and Cursive

22

Computer Problems in English

23

Pencept Commercial Product 1980s

24

Part 2Shorthand in Pen-Centric

PDAs Famous Uses of Shorthand Historical Shorthand Alphabets Pen-Centric Shorthand Alphabets Pen-Centric Word/Phrase Shorthand Allegro/Chatroom Shorthand System

M.S. thesis that could be extended into a DPS dissertation

25

Background Famous writings throughout history were

effectively written in a style of shorthand Cicero’s orations Martin Luther’s sermons Shakespeare’s and George Bernard Shaw’s

plays Samuel Pepys’ diary Sir Isaac Newton’s notebooks

26

Historical Shorthand Alphabets

We first review the history of shorthand systems prior to pen computing

Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases”

Shorthand can be traced back to the Greeks in 400 B.C.

27

Historical Shorthand Alphabets

We focus on shorthand alphabets that might be appropriate for PDAs

We review two types of shorthand Geometric shorthand

Small number of basic shapes Shapes reused in multiple orientations

Non-geometric shorthand shorthand

28

Historical Shorthand Alphabets

Ancient Greeks – 400 BC Tironian Alphabet – 63 BC John Willis’s Stenography – 1602 Gabelsberger Alphabet – 1834 Moon Alphabet – 1845

29

Tironian Alphabet, 63 B.C.Non-Geometric

30

Stenography Alphabet, 1602

31

Stenography Alphabet, 1602 Basic Shapes and Orientations

32

Gabelsberger Cursive-Style, 1834

Non-Geometric Alphabet

33

Moon Geometric Alphabet, 1845

34

Other Historical Shorthand Systems

Phonetic alphabets Pitman (1837), was popular in UK Gregg (1888), was popular in USA

Systems for the blind Braille (1821)

35

Pen-Centric Shorthand Alphabets

Some of the earliest were for CAD/CAM symbols represent graphical items and

commands Others developed for text input on small

consumer devices like PDAs that have limited computing power

We review geometric and non-geometric shorthands appropriate for small devices

36

Pen-Centric Shorthand Alphabets

Historical alphabets presented above could be used for machine recognition symbols drawn with a single stroke (except “K” in Tironian and “+” in

Stenography) In addition to shape and orientation,

online systems can use stroke direction to differentiate among symbols

37

Pen-Centric Shorthand Alphabets

Geometric Pen-Centric Shorthands Organek – 1991 Allen – filed 1991, patent 1993 Goldberg (Xerox) – filed 1993, patent 1997

Non-Geometric Pen-Centric Shorthands Graffiti (Palm Computing) – 1995 Allegro (Papyrus) – 1995

38

Organek Alphabet, 1991

39

Organic Alphabet, 1991 Basic Shapes and

Orientations

One shape in 4 orientations.This gives 8 directions that together with 3 lengths provide 24 symbols.A second wheel provides additional symbols.

40

Allen patent, filed 1991

41

Allen patent, filed 1991 Basic Shapes and

Orientations

42

Goldberg patent, filed 1993(“unistroke symbols”)

43

Goldberg patent, filed 1993 Basic Shapes and

Orientations

44

Goldberg patent, filed 19935 Basic shapes4 Orientations

2 Stroked Directions40 Possible Symbols

Designed for Speed of Input and Maximum Symbol Separation

45

Shorthand Alphabet Design

How would you design a shorthand alphabet?

What would be the design criteria?

46

Design of Graffiti Alphabetfor the Palm Pilot

Small alphabet Uppercase, digits, special symbols

One stroke per symbol to avoid segmentation difficulty

Separate writing areas for letters and digits to avoid same-shape confusions

47

Graffiti Alphabet, 1995

48

Graffiti Mimics Keyboard Input

Character by character input Mode shifts for

Uppercase Special characters

Eyes can focus on application’s insertion point rather than on input area

49

Graffiti Alphabet Design

What was the additional design criterion?

50

Graffiti Alphabet Design Designed for ease of learning

20 letters exactly match the Roman alphabet

6 remaining ones match partially

51

Graffiti Alphabet: 11 of 26 characters

have alternate variations

52

Frequently Confused Characters

53

Other Low Performance Characters

54

Symbol Overlap Comparison

55

Graffiti Recognition Accuracy Study

56

Allegro Alphabet (Papyrus), 1995 (now Microsoft)

57

Simplified Design Tradeoffs/Decisions

for Graffiti and Allegro PDA Alphabets

Small alphabet one case rather than both upper and lowercase

One stroke per character (character = stroke) allows machine to recognize each character upon pen lift

Small number of writing variations per letter preferably only one

Separate writing areas for letters and digits avoids confusion of similarly shaped letters and digits

High correspondence to Roman alphabet for ease of learning

non-geometric, might not actually qualify as shorthand

58

Commercially Successful Shorthands

Similar to the Roman alphabet Easy to learn Graffiti used in Palm OS devices

notably the Palm Pilot & Handspring models Allegro used in Microsoft Windows devices

Geometric alphabets not successful

59

Current Commercial SystemsCompany/System Writing Style Palm Computing/Graffiti* Special Shorthand Alphabet

Microsoft/Papyrus Allegro Special Shorthand Alphabet

CIC/Jot Relatively Unconstrained Handprint

Microsoft Relatively Unconstrained Handprint and Cursive

*A few years ago Palm switched from Graffiti to Graffiti2, Graffiti2 is basically Jot licensed from CIC.

60

Jeff Hawkins, innovator 1979 BSEE Cornell, 1979-1986 worked at Intel and GRiD 1986-1987 ABD BioPhysics doctoral program, U.C. Berkeley 1987- back at GRiD he created GRiDPAD, first pen computer 1992 formed Palm Computing, 1993 created first handwriting reco

software product for a mobile handheld - Casio’s Zoomer 1995 Palm Computing bought by U.S. Robotics 1996 created PalmPilot, first PDA with Graffiti shorthand alphabet

(over a million shipped in 18 months, a 66% market share, and the fastest growth of any computing product in history, faster than the TV and the VCR)

1997 U.S. Robotics bought by 3Com (sued by Xerox for patent infringement)

1998 left Palm to form Handspring, 1999 launched the Visor handheld 2000 Palm Computing spun off by 3Com 2002 created what is now the

Redwood Center for Theoretical Neuroscience 2003 Handspring (with Hawkins, et al.) acquired by Palm Computing 2005 Founded Numenta to build the ultimate brain-like machine

61

Palm-XeroxPatent Infringement Lawsuit

The nine-year old battle between Palm and Xerox over handwriting recognition ends in 2006, see article.

Palm pays Xerox $22.5 million for a fully paid-up license for Xerox patents covering its text input Unistrokes technology

Xerox first sued Palm predecessor Palm Computing back in April 1997, claiming that the Graffiti text-entry system used in its PDAs infringed on patents for Unistrokes, which allows users to input letters and numbers into personal data units with basic, one stroke movements.

62

ConclusionsPalm-Xerox Patent Infringement

Lawsuit Invalidity

Historical research showed that Goldberg alphabet not so unique

Even though the patent was accepted as valid, these arguments narrowed the scope of the patent

Infringement Analyses and comparisons of the Goldberg and

Graffiti alphabets showed major differences Result was favorable settlement for Palm

63

Pen-Centric Word/Phrase Shorthand

such as Chatroom Shorthand

Further increase speed of text entry

Potential applications Where input speed important Where word/phrase abbreviations

occur frequently – e.g., email

64

Chatroom Shorthand Examples

CU See you, or Cracking upCM Call me@TEOTD At the end of the day^5 High five2nite TonightLOL Laughing out loudASAP As soon as possibleB/C or BC Because

65

Allegro/Chatroom Shorthand System

Developed for M.S. dissertation Student was hearing impaired Developed as output component of

communication system Handwriting to text to speech

Two input writing areas One for Allegro (all-purpose) One for chatroom-like or user-defined

words/phrases

66

Allegro/Chatroom Shorthand System

Stroke acquisition GUI

allegro strokerecognition

alphabet

sentence accumulator

Sentence display and spoken output

allegro strokelibrary

user-defined stroke library

a single stroke

other strokerecognition

word/phrasecharacter

done? no

yes

meaning

is it

67

Allegro/Chatroom Shorthand System

68

Allegro/Chatroom Shorthand System

M.S. Thesis Experimental Results

Allegro/Chatroom pen-centric shorthand input is faster than typing text and is comparable to typing text and chatroom shorthand characters

69

ConclusionsPen-Centric Shorthands

Pen-centric interfaces should use shorthand, and especially word/phrase shorthand where appropriate, for fast text input

Benefit of shorthand interfaces Provides critical infrastructure for many pen-

centric applications Enhances natural pen-centric interactions for

many applications Has greatest impact on the utility of applications

running on small mobile devices

70

ConclusionsHandwriting Recognition

Graffiti and Allegro greatly simplified the recognition problem

Handprint problem not completely solved Even with IBM’s ThinkWrite, CIC’s Jot, and

Microsoft products Cursive script problem clearly not

solved

71

References W.B. Huber, S.-H. Cha, C.C. Tappert, and V.L. Hanson, "Use of

Chatroom Abbreviations and Shorthand Symbols in Pen Computing," Proc. 9th Int. Workshop on Frontiers in Handwriting Recognition, IWFHR 2004, Tokyo, Japan, October 2004.

W. Huber, V. Hanson, S. Cha, and C.C. Tappert, "Common Chatroom Abbreviations Speed Pen Computing," Proc. 11th Int. Conf. Human-Computer Interaction, Las Vegas, NV, July 2005.

C.C. Tappert and S. Cha, "Handwriting Recognition Interfaces," Chapter 6, pp. 123-137, in Text Entry Systems, Scott MacKenzie and Kumiko Tanaka-Ishii (Eds.), Morgan Kaufmann, 2007.

C.C. Tappert, C.Y. Suen, and T. Wakahara, "The state-of-the-art in on-line handwriting recognition," IEEE Trans. Pattern Analysis Machine Intelligence, Vol. PAMI-12, pp. 787-808, August 1990.

C.C. Tappert and J.R. Ward, "Pen-Centric Shorthand Handwriting Recognition Interfaces," Proc. 1st Int. Workshop on Pen-Based Learning Technologies, Catania, Italy, May 2007.