shorthand handwriting recognition for pen-centric interfaces

31
PLT 2007 CSIS CSIS Shorthand Handwriting Recognition for Pen-Centric Interfaces Charles C. Tappert 1 and Jean R. Ward 2 1 School of CSIS, Pace University, New York, USA 2 Pen Computing Consultant, Massachusetts, USA

Upload: jolie

Post on 13-Feb-2016

61 views

Category:

Documents


1 download

DESCRIPTION

Shorthand Handwriting Recognition for Pen-Centric Interfaces. Charles C. Tappert 1 and Jean R. Ward 2. 1 School of CSIS, Pace University, New York, USA 2 Pen Computing Consultant, Massachusetts, USA. Will provide critical infrastructure for many pen-centric applications - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Shorthand Handwriting Recognition for Pen-Centric Interfaces

Charles C. Tappert1 and Jean R. Ward2

1 School of CSIS, Pace University, New York, USA2 Pen Computing Consultant, Massachusetts, USA

Page 2: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Thesis: Pen-Centric, Chatroom-Like Shorthand Interfaces

• Will provide critical infrastructure for many pen-centric applications

• Will provide fast text input• Will have greatest impact on applications

running on small mobile devices

Page 3: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Agenda• Handwriting

– Fundamental Property of Writing– Handwriting Recognition Difficulties

• Historical Shorthand Alphabets• Online (Pen-Centric) Handwriting Recognition

– Online more accurate than Offline Recognition– Online Info Can Complicate Recognition Process– Design Tradeoffs/Decisions

• Pen-Centric Shorthand Alphabets• Pen-Centric Word/Phrase Shorthand• Allegro/Chatroom Experimental Shorthand System

Page 4: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Fundamental Property of Writing

• Differences between different characters are more significant than differences between different drawings of the same character

• This makes handwritten communication possible

Page 5: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Fundamental Property of Writing

• Property holds within subalphabets of uppercase, lowercase, and digits, but not across them

• “I”, “l”, and “1” written with single vertical stroke• “O” and “0” written similarly with an oval

Page 6: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Handwriting Recognition Difficulties

• Shape, size, and slant variation• Similarly shaped characters – U and V • Careless writing

– in the extreme, almost illegible writing• Resolving difficult ambiguities requires

sophisticated recognition algorithms, syntax/semantics

Page 7: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Historical Shorthand Alphabets(prior to pen computing)

• Famous writings were written in shorthand – Cicero’s orations– Martin Luther’s sermons– Shakespeare’s and George Bernard Shaw’s plays

• We focus on shorthand appropriate for PDAs• Two main types of shorthand

– Non-geometric shorthand– Geometric shorthand

• Small number of basic shapes• Shapes reused in multiple orientations

Page 8: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Tironian Alphabet, 63 B.C.

Page 9: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Stenographie Alphabet, 1602

Page 10: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Stenographie Alphabet, 1602

Geometric shorthand – basic shapes/orientations

Page 11: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Moon Alphabet, 1894• Geometric shorthand – basic shapes/orientations

Page 12: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Other Historical Shorthand Systems

• Phonetic alphabets– Pitman (1837)– Gregg (1885)

• Systems for the blind – Braille (1824)

• Cursive shorthands – Gabelsberger (1834)

Page 13: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Online (Pen-Centric) Handwriting Recognition

• Machine recognizes the writing as the user writes • Digitizer equipment captures the dynamic

information of the writing – Stroke number, order, direction, speed – A stroke is the writing from pen down to pen up

Page 14: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Online (Pen-Centric) more accurate than Offline (Static) Recognition

• Can use both dynamic and static information• Can often distinguish between similarly

shaped characters – E.g., 5 versus S where the 5 is usually written

with two strokes and the S with one stroke

Page 15: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Online Information Can Complicate Recognition Process

• Segmentation ambiguities– Character-within-character problem – cl versus d

• Large number of possible variations – E can be written with one, two, three, or four strokes, and with various

stroke orders and directions – Four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)

Page 16: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Design Tradeoffs/Decisions

• No constraints on the user– Machine recognizes user's normal writing

• User severely constrained– Must write in particular style such as handprint – Must write strokes in particular order, direction,

and graphical specification• Simplest is one stroke per character, one stroke

direction, one shape

Page 17: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Pen-Centric Shorthand Alphabets

• Some of the earliest were for CAD/CAM• Others developed for text input on PDAs • We review geometric and non-geometric

shorthands appropriate for small devices• Historical alphabets presented above could be

used for machine recognition• In addition to shape and orientation, stroke

direction can differentiate among symbols

Page 18: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allen Alphabet

Page 19: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allen AlphabetBasic Shapes and Orientations

Page 20: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Goldberg Alphabet

Page 21: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Goldberg AlphabetBasic Shapes and Orientations

Page 22: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Graffiti Alphabet(non geometric )

Page 23: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allegro Alphabet (non geometric)

Page 24: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Simplified Design Tradeoffs/Decisions for Graffiti and Allegro PDA Alphabets

• Small alphabet– one case rather than both upper and lowercase

• Small number of writing variations per letter– preferably only one

• One stroke per character (character = stroke)– allows machine to recognize each character upon pen lift

• Separate writing areas for letters and digits– avoids confusion of similarly shaped letters and digits

Page 25: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Graffiti and AllegroCommercially Successful Shorthands

• High correspondence to Roman alphabet– Easier to learn– Graffiti used in Palm OS devices

• notably the Palm Pilot and Handspring models – Allegro used in Microsoft Windows devices

• Geometric alphabets not successful

Page 26: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Pen-Centric Word/Phrase Shorthande.g., Chatroom Shorthand

• Further increase speed of text entry• Potential applications

– Where input speed important– Where word/phrase abbreviations occur

frequently – e.g., email

Page 27: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allegro/Chatroom Shorthand System

• Developed for M.S. dissertation– Student was hearing impaired– Developed as output component of

communication system• Handwriting to text to speech

• Two input writing areas– One for Allegro (all-purpose)– One for chatroom-like words/phrases (e.g., CUL, F2F)

Page 28: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allegro/Chatroom Shorthand SystemStroke acquisition GUI

allegro strokerecognition

alphabet

sentence accumulator

Sentence display and spoken output

allegro strokelibrary

user-defined stroke library

a single stroke

other strokerecognition

word/phrasecharacter

done? no

yes

meaning

is it

Page 29: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allegro/Chatroom Shorthand System

Page 30: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Allegro/Chatroom Shorthand System Preliminary Experimental Results

• Allegro/Chatroom pen-centric shorthand input faster than typing text and comparable to typing text and chatroom shorthand characters

Page 31: Shorthand Handwriting Recognition for Pen-Centric Interfaces

PLT 2007

CSISCSIS

Conclusions: Pen-Centric, Chatroom-Like Shorthand Interfaces

• Will provide critical infrastructure for many pen-centric applications

• Will provide fast text input• Will have greatest impact on applications

running on small mobile devices