1 pen-centric shorthand interfaces charles c. tappert seidenberg school of csis, pace university

1

Pen-CentricShorthand Interfaces

Charles C. TappertSeidenberg School of CSIS, Pace

University

2

Themes of Presentation

Online Handwriting Recognition and Pen Computing Tutorial

Historical Research – undertaken for the Palm-Xerox Patent Infringement Lawsuit

Recent Research - Enhanced Pen-Centric Shorthand Interfaces can have benefits DPS dissertation could extend M.S. thesis

3

Enhanced Pen-Centric Shorthand Interfaces

Can use word/phrase shorthand to speed text input

Can provide critical infrastructure for many pen-centric applications

Can enhance natural pen-centric interactions for many applications

Will have greatest impact on the utility of applications running on small mobile devices

4

Part 1: Online (Pen-Centric)Handwriting Recognition

Written Languages and Handwriting Properties The Fundamental Property of Writing Handwriting Recognition Difficulties Online (Pen-Centric) Handwriting Recognition Online more accurate than Offline Recognition Online Info Can Complicate Recognition

Process Design Tradeoffs / Design Decisions Computer Problems in English

5

Written Language and Handwriting Properties

Alphabet Letters, digits, punctuation, special

symbols Writing is a time sequence of strokes

Stroke – writing from pen down to pen up Usually complete one character before

beginning the next Spatial order – e.g., in English left to right

6

Fundamental Property of Writing

Differences between different characters are more significant than differences between different drawings of the same character

This makes handwritten communication possible

Can there be exceptions – say, different characters written identically?

7

Fundamental Property of Writing

in English

Property holds within subalphabets of uppercase, lowercase, and digits, but not across them

“I”, “l”, and “1” written with single vertical stroke

“O” and “0” written similarly with an oval

8

Handwriting Recognition Difficulties

Shape, size, and slant variation Similarly shaped characters – U and

V Careless writing

in the extreme, almost illegible writing Resolving difficult ambiguities

requires sophisticated recognition algorithms, syntax/semantics

9

Electronic tablets invented in late 1950s Digitizer and display in separate surfaces

Pen Computers arrived in 1980s Combined digitizer and display Brought input and output into one surface Immediate feedback via electronic ink Created paper-like interface

Online (Pen-Centric) Handwriting Recognition

10

Tablet Digitizers – Dynamic Information

Pen down – indication of inking X-Y coordinates as function of time

Sampling rate: 100 points per second Resolution: 200 points per inch

11

Early Pen-Centric Interface

Different surfaces for input and output

Rand system, about 1959

12

Pen Computers

IBM vision Paper-like interface,

1992

Microsoft Tablet PC Launched, 2001

13

Pen-Centric PDAs

Early Palm Pilot

Palm Tungsten T3 and Sony Clié TH55

14

Online (Pen-Centric) Handwriting Recognition

Machine recognizes the writing as the user writes

Digitizer equipment captures the dynamic information of the writing

Stroke number, order, direction, speed A stroke is the writing from pen down to pen up

15

Online (Pen-Centric) more accurate than Offline (Static)

Recognition

Can use both dynamic and static information

Can often distinguish between similarly shaped characters E.g., 5 versus S where the 5 is usually

written with two strokes and the S with one stroke

16

Online Information Can Complicate Recognition Process

Large number of possible variations E can be written with one, two, three, or four

strokes, and with various stroke orders and directions

A four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)

17

Online Information Can Complicate Recognition Process

Other variations

18

Online Information Can Complicate Recognition Process

Segmentation ambiguities character-within-character problem lowercase d might be recognized as a cl if

drawn with two strokes that are somewhat separated from one another

19

Design Tradeoffs/Decisions

No constraints on the user Machine recognizes user's normal

writing User severely constrained

Must write in particular style such as handprint

Must write strokes in particular order, direction, and graphical specification

20

English Writing Styles Handprint

Uppercase – about 2 strokes per letter Lowercase- about 1 stroke per letter

Cursive Script Usually less than 1 stroke per letter Delayed crossing and dotting strokes

21

Computer Problems in English

Constrained Handprint Printing one symbol per box – form filling Printing on lines – symbols can touch or

overlap Unconstrained Handprint

No lines and symbols can touch or overlap

Cursive Script Mixed Printing and Cursive

22

Computer Problems in English

23

Part 2Shorthand in Pen-Centric

PDAs

Famous Uses of Shorthand Historical Shorthand Alphabets Pen-Centric Shorthand Alphabets Pen-Centric Word/Phrase Shorthand Allegro/Chatroom Shorthand System

M.S. thesis that could be extended into a DPS dissertation

24

Background

Famous writings throughout history were effectively written in a style of shorthand Cicero’s orations Martin Luther’s sermons Shakespeare’s and George Bernard Shaw’s

plays Samuel Pepys’ diary Sir Isaac Newton’s notebooks

25

Historical Shorthand Alphabets

We first review the history of shorthand systems prior to pen computing

Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases”

Shorthand can be traced back to the Greeks in 400 B.C.

26

Historical Shorthand Alphabets

We focus on shorthand alphabets that might be appropriate for PDAs

We review two types of shorthand Geometric shorthand

Small number of basic shapes Shapes reused in multiple orientations

Non-geometric shorthand shorthand

27

Historical Shorthand Alphabets

Ancient Greeks – 400 BC Tironian Alphabet – 63 BC John Willis’s Stenography – 1602 Gabelsberger Alphabet – 1834 Moon Alphabet – 1845

28

Tironian Alphabet, 63 B.C.Non-Geometric

29

Stenography Alphabet, 1602

30

Stenography Alphabet, 1602

Basic Shapes and Orientations

31

Gabelsberger Cursive-Style, 1834

Non-Geometric Alphabet

32

Moon Geometric Alphabet, 1845

33

Other Historical Shorthand Systems

Phonetic alphabets Pitman (1837), was popular in UK Gregg (1888), was popular in USA

Systems for the blind Braille (1821)

34

Pen-Centric Shorthand Alphabets

Some of the earliest were for CAD/CAM symbols represent graphical items and

commands Others developed for text input on

small consumer devices like PDAs that have limited computing power

We review geometric and non-geometric shorthands appropriate for small devices

35

Pen-Centric Shorthand Alphabets

Historical alphabets presented above could be used for machine recognition symbols drawn with a single stroke (except “K” in Tironian and “+” in

Stenography) In addition to shape and orientation,

online systems can use stroke direction to differentiate among symbols

36

Pen-Centric Shorthand Alphabets

Geometric Pen-Centric Shorthands Organek – 1991 Allen – filed 1991, patent 1993 Goldberg (Xerox) – filed 1993, patent 1997

Non-Geometric Pen-Centric Shorthands Graffiti (Palm Computing) – 1995 Allegro (Papyrus) – 1995

37

Organek Alphabet, 1991

38

Organic Alphabet, 1991 Basic Shapes and

Orientations

One shape in 4 orientations.

This gives 8 directions that together with 3 lengths provide 24 symbols.

A second wheel provides additional symbols.

39

Allen patent, filed 1991

40

Allen patent, filed 1991 Basic Shapes and

Orientations

41

Goldberg patent, filed 1993(“unistroke symbols”)

42

Goldberg patent, filed 1993 Basic Shapes and

Orientations

43

Goldberg patent, filed 1993

5 Basic shapes4 Orientations

2 Stroked Directions40 Possible Symbols

Designed for Speed of Input and Maximum Symbol Separation

44

Shorthand Alphabet Design

How would you design a shorthand alphabet?

What would be the design criteria?

45

Design of Graffiti Alphabetfor the Palm Pilot

Small alphabet Uppercase, digits, special symbols

One stroke per symbol to avoid segmentation difficulty

Separate writing areas for letters and digits to avoid same-shape confusions

46

Graffiti Alphabet, 1995

47

Graffiti Mimics Keyboard Input

Character by character input Mode shifts for

Uppercase Special characters

Eyes can focus on application’s insertion point rather than on input area

48

Graffiti Alphabet Design

What was the additional design criterion?

49

Graffiti Alphabet Design

Designed for ease of learning 20 letters exactly match the Roman

alphabet 6 remaining ones match partially

50

Graffiti Alphabet: 11 of 26 characters

have alternate variations

51

Frequently Confused Characters

52

Other Low Performance Characters

53

Symbol Overlap Comparison

54

Graffiti Recognition Accuracy Study

55

Allegro Alphabet (Papyrus), 1995 (now Microsoft)

56

Simplified Design Tradeoffs/Decisions

for Graffiti and Allegro PDA Alphabets

Small alphabet one case rather than both upper and lowercase

One stroke per character (character = stroke) allows machine to recognize each character upon pen lift

Small number of writing variations per letter preferably only one

Separate writing areas for letters and digits avoids confusion of similarly shaped letters and digits

High correspondence to Roman alphabet for ease of learning

non-geometric, might not actually qualify as shorthand

57

Commercially Successful Shorthands

Similar to the Roman alphabet Easy to learn Graffiti used in Palm OS devices

notably the Palm Pilot & Handspring models Allegro used in Microsoft Windows devices

Geometric alphabets not successful

58

Current Commercial Systems

Company/System Writing Style

Palm Computing/Graffiti*

Special Shorthand Alphabet

Microsoft/Papyrus Allegro

Special Shorthand Alphabet

CIC/Jot Relatively Unconstrained Handprint

Microsoft Relatively Unconstrained Handprint and Cursive

*A few years ago Palm switched from Graffiti to Graffiti2, Graffiti2 is basically Jot licensed from CIC.

59

Jeff Hawkins, innovator 1979 BSEE Cornell, 1979-1986 worked at Intel and GRiD 1986-1987 ABD BioPhysics doctoral program, U.C. Berkeley 1987- back at GRiD he created GRiDPAD, first pen computer 1992 formed Palm Computing, 1993 created first handwriting reco

software product for a mobile handheld - Casio’s Zoomer 1995 Palm Computing bought by U.S. Robotics 1996 created PalmPilot, first PDA with Graffiti shorthand alphabet

(over a million shipped in 18 months, a 66% market share, and the fastest growth of any computing product in history, faster than the TV and the VCR)

1997 U.S. Robotics bought by 3Com (sued by Xerox for patent infringement)

1998 left Palm to form Handspring, 1999 launched the Visor handheld 2000 Palm Computing spun off by 3Com 2002 created what is now the

Redwood Center for Theoretical Neuroscience 2003 Handspring (with Hawkins, et al.) acquired by Palm Computing 2005 Founded Numenta to build the ultimate brain-like machine

60

Palm-XeroxPatent Infringement Lawsuit

The nine-year old battle between Palm and Xerox over handwriting recognition ends in 2006, see article.

Palm pays Xerox $22.5 million for a fully paid-up license for Xerox patents covering its text input Unistrokes technology

Xerox first sued Palm predecessor Palm Computing back in April 1997, claiming that the Graffiti text-entry system used in its PDAs infringed on patents for Unistrokes, which allows users to input letters and numbers into personal data units with basic, one stroke movements.

61

ConclusionsPalm-Xerox Patent Infringement

Lawsuit

Invalidity Historical research showed that Goldberg

alphabet not so unique Even though the patent was accepted as valid,

these arguments narrowed the scope of the patent

Infringement Analyses and comparisons of the Goldberg and

Graffiti alphabets showed major differences Result was favorable settlement for Palm

62

Pen-Centric Word/Phrase Shorthand

such as Chatroom Shorthand

Further increase speed of text entry

Potential applications Where input speed important Where word/phrase abbreviations

occur frequently – e.g., email

63

Chatroom Shorthand Examples

CU See you, or Cracking up

CM Call me

@TEOTD At the end of the day

^5 High five

2nite Tonight

LOL Laughing out loud

ASAP As soon as possible

B/C or BC Because

64

Allegro/Chatroom Shorthand System

Developed for M.S. dissertation Student was hearing impaired Developed as output component of

communication system Handwriting to text to speech

Two input writing areas One for Allegro (all-purpose) One for chatroom-like or user-defined

words/phrases

65

Allegro/Chatroom Shorthand System

Stroke acquisition GUI

allegro strokerecognition

alphabet

sentence accumulator

Sentence display and spoken output

allegro strokelibrary

user-defined stroke library

a single stroke

other strokerecognition

word/phrasecharacter

done?no

yes

meaning

is it

66

Allegro/Chatroom Shorthand System

67

Allegro/Chatroom Shorthand System

M.S. Thesis Experimental Results

Allegro/Chatroom pen-centric shorthand input is faster than typing text and is comparable to typing text and chatroom shorthand characters

68

ConclusionsPen-Centric Shorthands

Pen-centric interfaces should use shorthand, and especially word/phrase shorthand where appropriate, for fast text input

Benefit of shorthand interfaces Provides critical infrastructure for many pen-

centric applications Enhances natural pen-centric interactions for

many applications Has greatest impact on the utility of

applications running on small mobile devices

69

ConclusionsHandwriting Recognition

Graffiti and Allegro greatly simplified the recognition problem

Handprint problem not completely solved Even with IBM’s ThinkWrite, CIC’s Jot, and

Microsoft products Cursive script problem clearly not

solved

70

References W.B. Huber, S.-H. Cha, C.C. Tappert, and V.L. Hanson, "Use of

Chatroom Abbreviations and Shorthand Symbols in Pen Computing," Proc. 9th Int. Workshop on Frontiers in Handwriting Recognition, IWFHR 2004, Tokyo, Japan, October 2004.

W. Huber, V. Hanson, S. Cha, and C.C. Tappert, "Common Chatroom Abbreviations Speed Pen Computing," Proc. 11th Int. Conf. Human-Computer Interaction, Las Vegas, NV, July 2005.

C.C. Tappert and S. Cha, "Handwriting Recognition Interfaces," Chapter 6, pp. 123-137, in Text Entry Systems, Scott MacKenzie and Kumiko Tanaka-Ishii (Eds.), Morgan Kaufmann, 2007.

C.C. Tappert, C.Y. Suen, and T. Wakahara, "The state-of-the-art in on-line handwriting recognition," IEEE Trans. Pattern Analysis Machine Intelligence, Vol. PAMI-12, pp. 787-808, August 1990.

C.C. Tappert and J.R. Ward, "Pen-Centric Shorthand Handwriting Recognition Interfaces," Proc. 1st Int. Workshop on Pen-Based Learning Technologies, Catania, Italy, May 2007.