1.introduction - web viewwriters can access those model files stored on the server under their...

1Callaper

1.INTRODUCTIONThe ease of use provided by the classic pen-and-paper “interface” is unsurpassed. Except for writing long text passages, most of us regard it as the most convenient way of expressing, conveying, and storing one’s thoughts and ideas. However, pen and paper are hardly supported by current information technology. The integration of paper-based devices and modern computers has been hampered by two problems: the difficulty of recording and writing without affecting the natural look and feel of pen and paper, and the insufficient recognition rates provided by handwriting recognizers. However, considerable progress has been made over the recent years. While new and highly sophisticated pen-based hardware is announced almost monthly nowadays, handwriting recognition rates are steadily improving and will soon reach a level of common acceptance. These developments pave the way for a better integration of pen and paper into the daily workflow.

The main idea of the concept described in the following sections is to generate a so-called model file for all paper documents expecting handwritten input. A model file describes the structure of a document and provides the context knowledge necessary for handwriting recognition. Moreover, it contains information about how the recognized data should be processed, including its final destination. A unique ID printed on each document specifies the corresponding model file. Writers can access the model file of a document via the Internet, under the number specified on the document. In practice, this means calling the (phone) number of a server dispatching model files, which motivates the name of the concept “Callpaper” — a very transparent concept for the writer. The Callpaper concept nicely supports the simultaneous generation of both paper copies and electronic copies of the same document, without imposing any additional load on a writer. It thus introduces the benefits of information technology into paper based processes without affecting the traditional workflow. Needless to say, that handwriting recognition plays a crucial role here. However, the concept is independent of the hardware employed for capturing handwriting. The latest hardware devices are able to capture handwriting directly on paper, while retaining the look and feel of the common pen-and-paper interface. In fact, the latest hardware features enable a multi-dimensional paper space spanning many different features, such as writer position and writer identification.

We will be discussing about the advantages and disadvantages of paper, different hardware solutions for capturing handwriting on paper, a client-server implementation of the proposed concept, illustration of a typical application flow and the logical space spanned by modern hardware components.

Finally an overview of our current implementation status and a conclusion with an outlook on future plans.

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING NOVEMBER 2012

2Callaper

1.1.Pen and PaperWhat are the advantages and disadvantages of paper when compared to the devices of modern information technology? Though this seems to be an easy question, we must now admit that we have not answered it properly during the recent years, considering the fact that many of us were predicting an ever decreasing paper consumption caused by the technological progress. In fact, the advantages of paper have been grossly underestimated, in particular when it is utilized in combination with pens. Of course, the disadvantages of paper are obvious: limited feedback on user input, limited access and search functions, slow transfer et cetera. And yet, paper combines features that no other hardware device can offer. Paper has a high resolution and is thus pleasant to read. Paper can be easily folded and crushed. It is very cheap, a fact that provides us with a strikingly simple way of deleting data, namely disposing old paper and replacing it by new one. Paper supports fast input of both textual as well as graphical data. Moreover, it does not require tedious eye-hand coordination since the cursor is directly under the pen tip. Hence it puts less cognitive load on the user than the common mouse interface.

The limited feedback capability is probably the most important drawback of paper compared to hardware devices trying to mimic its features, such as high-resolution flat screens. This limited feedback capability requires different interface designs and calls for hybrid solutions with at least rudimentary visual or aural feedback.

2.RECORDING HANDWRITINGIt has long been possible to capture handwriting on paper by utilizing a standard digitizer lying beneath the paper. However, that is not quite what we want. We need a hardware device that allows us to retain the look and feel of paper we are so accustomed to. And, of course, we need on-line handwriting and not just a simple scan of an offline document. On-line information describes the dynamics of the writing process, providing important information for recognition and segmentation. The ideal device would be a pen capable of capturing handwriting, recognizing handwriting, and sending recognition results to the proper destination via a wireless Internet connection. While there is little doubt that this is technically possible, it will still take a while before we will see such functionality integrated in a small and handy device like a pen. Nevertheless, the current hardware realizations are already very impressive, though they are merely approximations of this ideal. What we will definitely see in the near future are hybrid systems that distribute their functions among several components. A mobile phone in combination with a pen attached to the phone via a wireless connection is such a hybrid solution for instance. While the pen captures the handwriting, the mobile phone transmits the (recognized) data over the Internet.

The following subsections describe the current technical approaches for capturing on-line handwriting on paper. Note that in order to capture on-line handwriting it is necessary to capture not only relative coordinates but the absolute position of the pen tip during writing. This is mainly because the system would otherwise lose the position of the pen once it is lifted from the paper (pen-up movement).This difficulty excludes some of the existing


3Callaper

standard tracking techniques, such as most of the techniques utilized for tracking mouse movements. Moreover, we must consider that paper itself is not a fixed medium but may also move while the pen writes or hovers over its surface.

2.1.Patterned paperAt the moment, the most flamboyant but also most controversial approach for capturing handwriting is patterned paper [1]. This approach is based on a barely perceptible pattern printed on all documents allowing handwritten input. The pattern serves two main purposes: First, it assigns a unique code to each pen position on the paper. Second, it allows an automatic identification of the paper, which in turn enables document specific handwriting processing. Technically, the pattern is a composition of combinatorial combinations of standard sub-patterns, which are constantly read by a small integrated pen-camera while writing on the

paper. While this technique offers many of the desired features that a modern pen-based device should possess, the requirement of non-standard paper is a major drawback. In fact, many people think that this restriction will eventually prevent those devices from becoming mainstream, omnipresent tools. Anyway, the concept presented in this paper (see Section 4 and 5) is an appealing alternative to this technique, offering the same features without the requirement of special paper.

2.2.Sensor clampThis method is based on an array of sensors attached directly to the paper [2]. The sensors receive signals emitted by the pen, while a processor utilizes those signals to calculate the pen’s exact global position. Currently, a combination of infrared and ultrasonic signals are typically used for this purpose. A sensor array attached firmly to a sheet of paper avoids the problem of paper moving under the pen tip. Also, it should be technologically possible to integrate the sensor array directly into the cap of the pen. The cap could then be easily removed from the pen and attached to the paper — a self-explanatory and handy interface. The concept presented in Section 4 and 5 fits in well here.

2.3.Paper detectorsThis is a relatively new method whose technical feasibility still needs to be proved, in particular, in combination with handwriting recognition [4]. The idea is to record handwriting without any external sensors outside the pen. Of course, this poses a very challenging technological problem. Up to the present day, no hardware device has been able to detect the absolute position of a pen on paper, with the accuracy needed by handwriting recognition, without the help of information received from outside the pen. The technical realization described in [4] utilizes lasers to capture the position of the pen and detect the boundaries of the paper. Its technical details are out of the scope of this paper. Nevertheless, the Callpaper concept presented in the following sections also harmonizes with this sophisticated approach.


4Callaper

3.CLIENT-SERVER ARCHITECTUREThe Callpaper concept is based on a client-server architecture that manages requests for context information essential for the proper recognition and processing of handwritten documents, independent of the writer’s location. Roughly speaking, the Callpaper framework provides a means to combine syntax and semantics over a long distance. Writer can enhance handwritten data with semantic information by establishing a connection over the Internet or local are a network to the Callpaper server. This basic principle allows automatic processing of handwritten documents without modifying the standard pen-and-paper interface. Figure1showsthearchitectureof the Callpaper concept, which comprises the following physical and logical components: Callpaper, Callpaper server, Callpaper client, and Callpaper model.

3.1.CallpaperCallpaper is the name of the paper that, in combination with a pen, can act as a computer terminal allowing us to interface computers with just pen and paper. Callpaper is by no means special paper. It is best described by the following handy formula:

Callpaper=Paper + ID + IT

In words, this formula states that every standard paper is a Callpaper as long as it has a unique ID assigned to it and a corresponding model file accessible over a network

(Information Technology). In particular, this definition implies that Callpaper is a physical object possessing all the advantages of paper.

3.2.Callpaper serverThe Callpaper server is basically a large database containing model files corresponding to Callpapers. Writers can access those model files stored on the server under their unique IDs. Upon receiving a request for a specific model, the server dispatches the model file to the client site, where it provides context information essential for handwriting recognition. Depending on the server’s implementation and the type of models, handwriting recognition may also be performed on the server side. In that case, the server does

not only receive requests for model files but also handwritten data on which it performs complex handwriting recognition tasks according to the model file. For the writer, a

request for a paper model is straightforward. He just needs to call a Callpaper’s unique ID to access the corresponding model file, which is then received by the Callpaper client over the Internet or LAN.

3.3.Callpaper clientThe Callpaper client is the software that manages the underlying communication with the Callpaper server on the client site. Accordingly, one important task is to download and interpret model files. Furthermore, the client recognizes handwriting on a document according to the specifications given in the document’s model file. It also processes the final recognition result and/or transmits it to the final destination, as specified in the model file.


5Callaper

The Callpaper client is implemented either directly in the pen, or in a mobile phone connected to the pen via a wireless connection, or in a desktop computer. However, given the current state-of-the-art, the phone/pen-combination is much smaller and cheaper to realize than a single pen device featuring the same functionality. Depending on the hardware realization, the client may offer a rudimentary user interface and support simple feedback with an LED or a speaker.

3.4.Callpaper modelCallpaper models describe the meaning and structure of a document and provide instructions for recognizing the handwritten information thereon. The data contained in the model file is especially needed for mapping captured points generated by a pen device from the physical space into the logical space of a document. It is this mapping that allows us to use standard paper for interfacing electronic devices. In addition to this, a model file contains context information about how a document’s recognized handwriting should be processed. Some informations a model file can provide are, for example:

Paper type: form, drawing paper, blank paper, ...

Paper access: paper validity, expirations, privacy, ...

Document fields: names, numbers, signatures, ...

Consistency checks: dates, sums, order numbers,..

Recipient address: email address, fax number, ...

Transmission type: email, fax, ftp, ...


6Callaper

Figure:3.1. Callpaper Architecture

4.APPLICATION FLOWThe Callpaper concept offers a strikingly simple way for a writer to enter handwritten, paper-based information into an electronic device, and thus to bridge the gap between paper world and electronic world. Let us assume that the writer uses a pen/phone hardware combination. The pen captures the handwriting utilizing one of the above mentioned techniques and is attached to the phone via a wireless connection, such as Bluetooth or infrared. Furthermore, let us assume that the client, i.e. the phone, is capable of performing all necessary handwriting recognition tasks. Then, for the writer, accessing Callpaper functionality is hardly more than a simple phone call. For instance, sending a handwritten Callpaper via email to a recipient consists of three simple steps:

1. Calling the server: First, the writer dials the paper ID printed on the paper to call the Callpaper server. The server searches the corresponding model file in its database and returns it promptly to the writer.

2. Writing the paper: In the second step, the writer proceeds as normal: He writes the document having the look and feel of normal paper, while the pen is simultaneously capturing his handwriting.

3. Sending the data: Finally, the handwriting is recognized and the recognition result send to the recipient according to the specifications given in the model file. The writer can trigger the recognition and transmission of handwriting by either writing a special gesture on a position


7Callaper

determined in the model file, such as a signature, or just by pressing a button of the mobile phone.

6. Multi-dimensional Callpaper space. A nice feature of the Callpaper concept is that paper is no longer confined to two dimensions. In combination with modern information technology, paper becomes a multidimensional space. For instance, a combination of pen and state-of-the-art cellular phone with integrated GPS receiver can support the following features:

Writer position (2-dimensions) +

Pen Position (2-dimensions) +

Callpaper ID (1-dimension) +

Writer ID (1-dimension) +

Pen ID (1-dimension) +

Time (1-dimension)

—————————————————

8-dimensional Callpaper space

These features expand the two-dimensional hyperspace of the writing plane into an eight dimensional space. This expanded space extends the possibilities of document analysis

and offers a wide range of new applications. It leads to a better customization of document processing, considering in more detail the specific requirements of customers in different locations and environments.

Alternatively, the digital pen can be configured to transmit pen status information to the host computer. In either case, the host computer can display the pen status information. Pen status information may include, for example, battery status, available memory, used memory, pen serial number, pen identifier, user settings (e.g., configuration of selection mechanism), and the like. Moreover, pen information may also include commands transmitted from the host computer to the digital pen 100, for example, setting an indicator (e.g., vibrator, LED) ON/OFF or setting power-off delays, or synchronizing a calendar, or the like.

The implementation of callpaper technology is using the patterned paper to record

handwriting since the last decade. In aiding the technique, a specialised pen called digital pen was introduced. The technology, extend of usage in the callpaper scenario of the digital pen is discussed next.


8Callaper

5.DIGITAL PEN

The present invention relates generally to digital writing input devices for computer systems and more particularly to a multi-function digital pen with function control features.

5.1.ABOUT DIGITAL PENConventional digital pen devices that capture handwriting strokes are known. One type of conventional prior art digital pen device includes a digital pen and tablet configuration. The digital pen and tablet couple together. The tablet couples with a personal computer. The tablet includes electronics to recognize character strokes when the digital pen is in contact with the tablet. In addition, the tablet may include special function areas on which a user may tap with the digital pen to invoke a function. The function may be pre-configured or pre-programmed. For example, the function can be pre-programmed to include a keystroke function such as open file, e.g., or print file, e.g., <CNTL><O>, or print file, e.g., <CNTL><P>

The digital pen and tablet configuration includes a number of drawbacks. For example, the digital pen and tablet are bulky units. They require a large surface area for use and are too heavy to be considered portable. In addition, the pen is not useful without the tablet and vice versa. To help address these shortcomings, another conventional digital pen device has appeared on the market, the Anoto Digital Pen, which is manufactured by a subsidiary of C Technologies AB of Sweden, Anoto AB of Sweden.

The Anoto Digital Pen includes a digital camera, an advanced image-processing unit and a Bluetooth radio transceiver. It also contains an ink cartridge so a user can see what is written or drawn. The image capturing mechanism of the Anoto Digital Pen is activated by removing a cap and deactivated by replacing the cap again. However, the image processing capability of the Anoto Digital Pen is operational only when the Anoto Digital Pen is used with a specially designed paper. The specially designed paper must include a custom pattern that consists of very small dots slightly displaced from a grid structure and a special transfer box (sometimes referred to as magic “boxes”). As a user writes or draws on the specially designed paper with the Anoto Digital Pen, the Anoto Digital Pen creates a digital trace of whatever is done. This information is stored in the pen until the transfer indication box is selected. It is then forwarded from your digital pen directly to your nearby personal computer, or by a Bluetooth TM device—such as a Bluetooth TM enabled mobile phone—to any computer, mobile phone or information bank in the world via the Internet.

One drawback of the Anoto Digital Pens and other types of digital pens that capture handwriting strokes is that the digital pens themselves lack functional buttons or displays to capture additional handwritten gestures such as highlighting, bolding, or color change. For example, when a user handwrites notes and then uses a highlighter to emphasize particular text or patterns, the conventional digital pen device or system such as those described above,


9Callaper

only capture the original handwritten information, but are unable to capture the emphasized text or pattern. Moreover, an attempt to go back and, for example, highlight text, using a conventional digital pen device or system may result in canceling or crossing out previously written text.

Some other drawbacks of some digital pens that capture handwriting strokes are that the digital pens do not have added functionalities which can make these pens useful in other ways to a user. For instance, a digital pen with an alarm functionality could be used by a user to remind herself of certain important appointments etc. Such notification to the user could be provided by sound (eg., a beep, a buzz, etc.), by vibration of the pen, and so on. Another example is a digital pen that can function as a calculator, which could be used to perform quick mathematical calculations. With such a digital pen, a user would be able to dispense with pocket calculators.

Yet another drawback of conventional digital pens is that they do not have a voice-recording functionality. Such a voice-recording functionality would be useful in several situations. For instance, such a feature would be very useful in instances when the user cannot write (e.g., when the user is driving a car), or when the user does not have any special paper available at hand.

Still another drawback of conventional digital pens is that the special paper and the “magic boxes” on such a special paper described above are not reusable, since ink adheres to the paper when the pen is used. In certain situations, it may be desirable to not have the ink adhere to the paper, so that the paper can then be reused. For instance, a user may desire to use over and over again, a “magic box” to transmit written material to a host.Therefore, there is a need for (1) a digital writing system that captures additional handwritten gestures in addition to handwritten text and patterns; (2) a digital writing system with additional functionalities; (3) a digital writing system with a voice recording and recognition capability; and (4) a digital writing system where, in certain situations, the ink does not adhere to the paper.

In accordance with one illustrative embodiment of the present invention, a writing system is configured for capturing image data associated with writing gestures. The writing system also provides additional formatting functionality, for example, bolding, highlighting, underlining, italicizing, and the like. Further, the writing system may include other features that conveniently complement the handwriting capturing functions, such as, for example, a digital camera, a pocket calculator, a user notification feature, alarms, timers, calendar function, biometric sensors, and the like. In one embodiment, the writing system according to the present invention includes a digital pen device with selection mechanisms for a user to select among the different alternative functions provided, for example, highlighting, bolding, underlining, and the like. According to another aspect of the present invention, it is provided


10Callaper

a user interface for the management of data files generated by a system according to one embodiment of the invention.

A method according to one embodiment of the present invention includes the capture of information relating to a user’s writing. For example, in response to selection signals, a method according to this embodiment includes formatting of captured information. The formatting may include bolding, underlining, highlighting, color change, among other features that may be applied to the captured handwriting. According to one embodiment of the present invention, the formatting features can be selected at diiferent times With respect to the capturing of the handwriting gestures, for example, a bolding selection can be made prior to capturing the handwriting text to be bolded, or alternatively, once some handwriting has been captured, it can be selected for applying highlighting to it.

In addition, according to another aspect of the present invention, several embodiments are presented with additional functions provided. For example, one embodiment of the present invention includes a notification feature that allows a user to preset a trigger or alarm for the notification to take place, such as, for example, a timer or a calendar entry. Several alternatives are also shown for the notification feature including an audible alarm, vibration, visual display or the like. Other convenient functions presented include still or video capturing devices, audio input and output devices, calculator, biometrics, and the like.

The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.

5.2. DESCRIPTION OF DIGIPENFIG. 1 is an illustration of a digital pen device according to one embodiment of the present invention.FIG. 2 is a functional block diagram of a digital pen device according to one embodiment of the present invention.FIG. 3 illustrates a sample function control subsystem software within a software stack for a digital pen device according to one embodiment of the present invention.FIG. 4 illustrates a sample process operation of a function control subsystem according to one embodiment of the present invention.FIGS. 5a, 5b, and 5c show an illustration of one example of an operation according to one embodiment of the present invention.FIG. 6a is sample screen shot of a user interface according to one embodiment of the present invention.


11Callaper

FIG. 6b is a sample detailed view of a screen shot of a user interface according to one embodiment of the present invention.

The Figures and the following description relate to preferred embodiments of the present invention by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein Will be readily recognized as viable alternatives that may be employed without departing from the principles of the claimed invention.

One embodiment of the present invention comprises a digital pen device that includes function control selection mechanisms, e.g., buttons, pressure switches, or the like. A user that is writing text, patterns, or other handwritten expression on a writing medium, e.g., paper, may operate the digital pen device. The digital pen device captures the handwritten expression, for example, as images, and stores the handwritten expression as data. If additional expression is added to the handwritten expression, e.g., highlighting a word, the function control buttons are configured to capture this additional expression as data too.

Figure:5.1 Digital Pen

FIG. 1 illustrates a digital pen device in accordance with one embodiment of the present invention. The digital pen device includes a pen barrel, an optional grip area, a pen (or ink) tip, a character recognition lens 125, a function control box and a display area. In one embodiment, the ink tip 120 and the character recognition lens are in close proximity to each other. In addition, the function control box may be comprised of one or more mechanical or electrical (e.g., touch sensitive or optical) switches. The function control box may be configured for placement along a side of the pen barrel.


12Callaper

Figure:5.2. Functional diagram of DigipenFIG. 2 illustrates a functional block diagram of a digital pen device in accordance with one embodiment of the present invention The digital pen device includes a character recognition subsystem 210, a function control subsystem 220, a processor (or controller) subsystem 230, and a memory subsystem 240. The character recognition subsystem 210 may include a character recognition lens 125, and may couple with the processor subsystem 230 and the memory subsystem 240. The function control subsystem 220 couples with the processor subsystem 230 and may also couple with the memory subsystem 240 and/or the character recognition subsystem 210.

The character recognition subsystem 210 may be a digital camera or may be an optical character recognition system that includes pattern recognition sensors, e.g., photosensors. In some embodiments, the digital pen includes a separate mini camera 290. In such embodiments, the digital pen can be used as a camera as Well as a pen, and can be used capture images or video sequences. In other embodiments, when the character recognition subsystem 210 is a camera, the character recognition subsystem 210 itself may be used to capture images or video sequences.

The character recognition subsystem 210 is configured to capture handwriting gestures, e.g., handwritten text, numbers, symbols, drawings, or other impressions that may appear on a written medium. The function control subsystem 220 may include dedicated (or pre-defined) or configurable (e.g., user definable) functional operations that may, for example, be used in


13Callaper

association With one or more application programs. For example, the function control subsystem 220 may be configured to identify an area that a user has highlighted using a highlighter and transmit that information back to an application associated with the character recognition subsystem 210. The processor subsystem 230 provides conventional processor functions, including fetching and executing instructions to carry out functions, for example, functions defined by the character recognition subsystem 210 and the function control subsystem 220. The memory subsystem 240 may include volatile memory (e.g., dynamic and/or static random access memory) and/or non-volatile memory (e.g., flash memory and/or electrically erasable programmable read only memory).

The digital pen device may also include an optional ink cartridge, configured in close proximity to the character recognition device, so that impression written by a user is immediately picked up by the character recognition subsystem 210. In addition, the digital pen device may include a communication interface subsystem. The communication interface subsystem may be for wired (tethered) or wireless (non-tethered) communication. The communication interface subsystem may be, for example, a radio frequency transceiver or a universal serial bus interface. Alternatively, the communication interface subsystem may be an Infra Red communication system. The communication interface subsystem couples with, for example, the processor subsystem 230 to transmit information to, for example, a host computer in either real-time or batch mode (e. g., synchronization process such as a docking connection or universal serial bus connection on the pen, or the like). It is also noted that the digital pen device components discussed above may all couple together via a data bus in the digital pen device.


14Callaper

Figure:5.3. Functional Control subsystem

FIG. 3 illustrates one embodiment of the function control subsystem 220 software within a software stack for a digital pen device in accordance with one embodiment of the present invention The software stack includes a pen device driver 310 and an application 320. The software 220a for the function control subsystem 220 can reside on top of the pen device driver 310, an operating system, or within each individually or in combination. The pen device driver 220 couples with the data bus . The pen data bus may include an electrical connection with the function control box 130 (or switches 130a-130c). According to one embodiment of the present invention, a user may configure the selection mechanisms 130a- 130c, to correspond to a particular function, e.g., highlighting, bolding, italicizing, underlining, changing font color, or the like. The configuration may be provided on the digital pen itself, or may be done, for example, via a software application on a personal computer that is transmitted to the digital pen. Further, different users may configure the selection mechanisms differently and a set of user profiles can be stored on the digital pen and selected prior to operation, for example, using biometrics as discussed below.

During operation, when the user triggers a selection mechanism, e.g., any of 130a-1300 alone or in combination, the selection or switching mechanisms send a function selection signal back to the pen data bus to the processor subsystem 230, to the pen device control, and then to the function control subsystem 220. The function control subsystem 220, interprets the function selected and signals the operating system or the application 320 regarding the selected function. The handwriting gestures with which the formatting is associated can be determined in several ways. For example, a user can actuate a selection mechanism to select a formatting function prior to handwriting the gestures to which the formatting applies. Once finished with the handwriting, the user may actuate the same or another selection mechanism to indicate the end of the handwriting gestures to which the formatting applies. In an alternative embodiment, the user may produce the handwriting gestures first, then select a subset of handwriting gestures, and actuate a selection mechanism to apply the formatting to the selected handwriting gestures.

According to the present invention, there are several possible techniques for selecting a subset of handwriting gestures in order to apply the formatting function to it. For example, in one embodiment, patterned paper comprising machine-readable marks or “glyphs” provides a coordinate system that can be interpreted to uniquely identify the location of each individual glyph. This coordinate or location information is typically captured along with the handwriting gestures. Subsequently, a user may indicate the coordinate information (e.g., selecting a set of glyphs) of two opposite corners of a rectangular area on the patterned paper and using the captured coordinate information the processor can determine the subset of handwriting gestures located within the selected rectangular area.


15Callaper

Figure:5.5. Initiating handwriting recording

For example, FIGS. 5a through 5c illustrate one example of an operation in accordance with one embodiment of the present invention. In this example, existing handwriting gestures are selected by indicating the coordinates of the top left beginning of a handwriting subset and the coordinates of the bottom right end of the subset. Then, the user actuates function control switch 130a, configured for example to “highlight” handwriting gestures. The digital pen then transmits the function (“highlight”) and coordinate information back to the processor subsystem 230, onto the pen device driver 310 and the function control subsystem 220. The function control subsystem 220 would provide information to the application 320 that tells it to highlight the selected area provided by the position coordinates.

According to another embodiment of the present invention, the selection of a subset of captured handwriting gestures for applying a selected formatting function to includes an image processing function. According to this embodiment, handwriting gestures may be captured using an optical imaging system, for example, a photosensor array. The handwriting gestures are captured as bitmaps or similar images created based on the light reflected from a writing surface. Subsequently, a user may actuate a selection mechanism, e.g. a button, and recapture a subset of handwriting gestures to which a formatting function should be applied. Then, a processor can compare previously captured images with the currently captured images to find a match thereby determining the subset of captured handwriting gestures to which the selected formatting function applies. Several other implementations are possible to


16Callaper

provide the same functionality. For example, rather than using a selection mechanism, a user may change the tip of the digital pen on a mutiple-tip pen embodiment from a ball pen tip, to a highlighter tip and the digital pen Would automatically recognize the highlighting function from the use of the hig1ighter tip and capture the highlighting information accordingly.

In another embodiment, it is possible that a single selection mechanism 130a can be pressed to perform various distinct functions, based on the level or pressure/force applied. For instance, a slight pressure applied to control switch 130a might result in highlighting of the text, some more pressure applied to control switch 130a might result in bolding of the text, while still more pressure applied to control switch 130a might result in changing the color of the text. In one embodiment, the pressure applied by the user can be sensed using a Force Sensitive Resistor (“FSR”) located at the tip of the digital pen. In this embodiment, it is not necessary to have a specifically dedicated button on the pen for each desired functionality. In an alternative embodiment, the selection mechanism can be configured as an n-way switch, where n is any integer and corresponds to the number of states (e.g., functions that may be available). Another example is that in one embodiment, a function control switch 130a is configured for voice recording. In such an embodiment, an apparatus in accordance with the present invention is a dual mode pocket memo recorder that both Written- and voice-memo download to a host. Such a digital pen can be used, for example, to record to-do list items in the digital pen’s memory. Moreover, such a digital pen can be especially useful in situations when a user cannot write, or when the user has no special paper available at hand. For example, a user may be driving a car.Voice recording by the digital pen can have several other benefits as well. For example, in one embodiment of the present invention, voice recording is linked to written material. For instance, a user could draw on paper using a digital pen, and record a verbal description of the drawing. Such a verbal description, along with the linked drawing would provide for a very accurate and clear capture of the thoughts of the user. Yet another example of a benefit of voice recording is that in one embodiment of the present invention, voice recording is used in conjunction With voice recognition. Such a combination can be used, for instance, for email address look-up.

In such an embodiment, the digital pen includes a Coder-Decoder (“CODEC”) to perform functions such as converting a voice analog signal into a digital signal. The CODEC can be implemented in hardware, software, or firmware, or in any combination of these. In one embodiment, compression can be used to store the digital voice data in its stand alone memory (non-volatile) in a more effective way, in order to implement a longer recording time. In one embodiment, the compression algorithm used is compatible with PC standards.

In one embodiment of the present invention, a voice-to-text conversion module 270 is also present in the digital pen. The voice-to-text conversion module 270 can be used to convert


17Callaper

voice data into text with a specific voice recognition software application, as available today with some voice recorders.

In one embodiment, the digital pen may also have an audio input and an audio output. The audio output could be, for example, a speaker or earphonejack. Both the audio input and the audio output maybe processed by a CODEC. Such a digital pen can be used as a stand alone voice recording and playback system, i.e., without necessarily downloading to a host.

In one embodiment of the present invention, a 3iometric sensor module 280 is also present in the digital pen. For instance this biometric sensor module could be a fingerprint sensor. Such a biometric sensor module can be used for identification of the user of the pen. For some applications, this could be in lieu of a password. If multiple users may be using a digital pen, in some embodiments, such biometric sensing could also be used to retrieve each user’s preferred settings when that user starts using the pen. In some embodiments, such preferences may also be associated automatically with a user, by identifying the user via biometric sensing. It is noted that the biometric sensor in one embodiment may be functional with the optional grip area or in lieu of the optional grip area.

Another example of a functionality that can be available in the digital pen is that of a pocket calculator. Such a digital pen would provide the user with a “pocket calculator” which is easily available, and is small and convenient to carry.

In one embodiment, the numbers and the operators could be captured by handwriting. Thus no keyboard would be needed. In another embodiment, the capture may also be implemented by using special paper with printed numbers and operators. Further, in one embodiment, the result of the calculation could be shown on an on-board display 135, e.g., a liquid crystal display (“LCD”).

Yet another example of a functionality that can be available in the digital pen is that of a user notification function, e. g., an alarm notification or the like, provided in response to a user programmed triggering event. The special paper may include conventional patterned paper, e.g., having glyphs, and the “magic boxes” may be pre-set patterns corresponding to a function once that pattern image is detected. In addition, in one embodiment, special paper with specific fields (“magic boxes”) can be used to enter an alarm. In one embodiment, different triggering events for the user notifications can be pre-programmed depending on the functionality available in the digital pen. For example, in a digital pen equipped with a digital calendar feature, the triggering event may be an appointment at a certain date and time or a reminder for a special occasion, or the like. Alternatively, in a digital pen that includes a digital Watch or a digital timer, the triggering event may be a timer/countdown alarm, or a wake-up alarm, or the like. The user notification can take place in several different ways. For


18Callaper

example, some possible user notifications include without limitation, an alarm via the display 135, a buzzer (sound feedback), a vibration of the digital pen 100, or via one or more light emitting diodes (“LEDs”).

In one embodiment with a digital pen having a digital calendar feature, based on the appointments, meetings, reminders, or the like, entered with the digital pen 100, the calendar could be synchronized between the digital pen and a host computer. For example, the synchronizing may be done using the communication module of the digital pen either automatically or at user selected intervals.

Figure:5.4. Process operation of Function Control Subsystem

FIG. 4 illustrates a process operation of the function control subsystem 220 in accordance with one embodiment of the present invention. The process starts 410 and a particular function is selected 420 through a selection mechanism, e.g. 130a. A position on a writing surface, for example, a piece of paper, is identified 430 on which the functional operation is to occur. In one embodiment, this includes identifying a first location, p1, and a second location, p2 (see, e.g., FIG. 5b) by moving the digital pen over or on those two areas to trigger a selection mechanism (eg, a switch or a sensor). The function control subsystem 220 then calculates 440 an area, A, using the two position data points. This information may then be passed to, for example, the application 320 to apply the function, e.g., highlighting previously written text as shown in FIGS. 5a through 5c. The process then ends 460 by shutting off or waiting for the next function operation.A writing system in accordance with one embodiment of the present invention may also be


19Callaper

used with a special paper such that ink does not adhere to the paper. Such non-adherence of the ink could result in reusability. In one embodiment, a card which fits into a wallet (e.g., sized like a credit card) could be carried by a user. The user could check of a “magic box” on this card every time he Wanted to perform a specific function (e.g., transfer to the host data written on special paper, transfer to the host data written only on a specified page, etc.). In one embodiment, the special paper on such a card could be treated such that ink will not adhere to it.

In one embodiment, the “magic boxes” are predefined and may be Intelligent Character Recognition (“ICR”) fields. Thus, for example, in order to transfer data written on a specified page, the page number could be “Written” into an ICR field on the card. In another embodiment, a visual keyboard could be visible on the card, and the appropriate “key” could be tapped on in order to perform a specific function. This could be an alternative to using ICR. Another alternative to using ICR is the use of graffiti, as is well-known in the context of Personal Digital Assistants (“PDAS”). In an alternative embodiment, the “magic box” may be a predetermined pattern (e.g., a character) that is assigned a particular function or operation such that it is performed once the optical system recognizes the predetermined pattern.

Yet another use of the magic boxes is for purposes of “speed-mail.” It is to be noted that in some embodiments, such magic boxes are on the card described above. In other embodiments, the magic boxes are on the paper on which the user is writing. In addition, in some embodiments, ink may adhere to such magic boxes, while in other embodiments, ink may not adhere to such magic boxes. The speed-mail notion is analogous to the speed-dial feature available on most telephone instruments today. Specific e-mail addresses can be pre-programmed into specific numbers (e.g., “1” corresponds to mom’s email address, etc.), and then the specific number could be written into a magic box to send the data to the specified email address.

Figure:5.6.A. Storing document onto a system.


20Callaper

Once documents are created using the digital pen, and these are transferred to a host, it should be easy for a user to find them and use them. FIG. 6a is a screenshot of an embodiment of a user interface 600 on a host, which can make it easier for users to find, view, and integrate digital writing documents into other applications.

As can be seen from FIG. 6a, in one embodiment, all documents created with the pen are listed under a folder entitled “My Pen Documents.” It is to be noted that in one embodiment, this destination folder can be renamed or changed by the user as desired. The created documents can be sorted by different criteria (e.g., date, type, etc.). Further, this user interface provides for a preview of the various documents without the need for any third party software. It is also possible to search the documents using a search form.

Figure:5.6.B. User interface for accessing documents

FIG. 6b provides a more detailed view of the user interface 600. In one embodiment, the user interface 600 allows the user to use the documents directly (e.g., print them, send as an attachment in an email, send directly in an email, etc.). Moreover, when the documents are transferred to the host, some tags may be inserted into them (e.g., date, time, author’s name, sub-title, keyword etc.). In one embodiment, the insertion of such tags may be automatic. In another embodiment, the insertion of such tags may be manual. In yet another embodiment, the insertion of such tags may be a combination of automatic and manual.

In one embodiment, the user interface may include a list of templates for each type of document. (The various “types” of documents can include emails, faxes, letters, meeting minutes, notes, to-do lists, drawings, calendar entries, and so on). For instance, if the document is a fax document, the user can automatically insert the document into a fax


21Callaper

template. The document can them be faxed using third party software. In one embodiment, the user interface permits the user to seamlessly integrate with third party software.

In one embodiment, the user may be able to assign certain rules to a specific document, or to a type of document. For instance, if a document is an email, a user can assign rules such as which application to use for the email (e.g., Microsoft Outlook TM, Lotus Notes TM, etc.), the way the document Will appear (e.g., in draft form, etc.), and so on. Further, such rules can be assigned to the specific email in question, or to all emails created with the digital pen. Moreover, a rule may be set as a permanent rule or as a temporary rule.

In one embodiment of the present invention, a user can set up such rules by using a “Rule-setup Wizard” which assists the user in defining and assigning rules.

The digital pen may also be configured to provide bi-directional (or alternatively unidirectional) communication with another device, e.g., a host computer, through its communication subsystem. For example, in one embodiment when the digital pen is connected with a host computer, the host computer can be configured to retrieve digital pen status information.

6.IMPLEMENTATIONOur strategy is to start with simple documents requiring manageable handwriting recognition or none at all and then extend to more complex papers. Oddly enough, blank paper has the highest complexity in this respect since it provides the least context information for handwriting recognition. Accordingly, the first implemented Callpapers have been simple multiple choice tests. Figure6.1. shows a simple model file corresponding to a multiple choice document whose ticked answers are send by email to a collector. The syntax of this model file is akin to standard HTML or SGML syntax.

Figure:6.1. A model file for multiple choice document

The first line specifies the paper type and other essential attributes, such as the address of the recipient and the type of transaction. The attribute “Multiple” specifies whether multiple ticks are allowed for this multiple choice


22Callaper

test or not. This allows consistency checks by the Callpaper client later on. Moreover, the first line contains the coordinates of four calibration points necessary for calibrating the

pen’s coordinate space so that it matches the logical paper space. A calibration mode, which is automatically started directly after downloading the model file, requires the writer to tap four points with the pen tip, one in each corner of the paper. The captured points are then matched with the model points, utilizing a linear normalization of translation, rotation, and scaling. Mathematically, only three points are needed for this purpose and the fourth point serves just as a control point reducing the effect of imprecise user input. The main body of the model file in Figure 2 is a list of tick field specifications. Each specification describes the upper left and lower right point of a tick field’s bounding box in coordinates of the logical paper space. A return string at the end of each specification contains the text that is send to the recipient via e-mail in case the field is ticked by the writer. The last line of the model file specifies the so-called action field. By simply ticking this action field, the writer can trigger the processing of handwriting and the transmission of its recognition result to the recipient. In the present client-server setup, the client performs all processing and handwriting recognition, while the server acts mainly as a file server for model files. At present, the client runs on a desktop computer. However, it offers merely basic interface functions, such as simple acoustic feedback, to emulate its application in a small hand-held device like a pen or a phone.

7.APPLICATIONS

7.1.Business solutions

7.1.1.Health careWith a digital pen, care workers no longer need to manually type in data, but they can spend more time interacting with patients instead. Moreover, the pen is a socially accepted tool that doesn’t disrupt face-to-face interaction. Here are some examples of what the digital pen has done in the field of healthcare:

It helps the UN to prevent the outbreak of animal diseases in rural Africa. At a private healthcare practice, 10-12 additional patients can be received by each doctor every day. It has streamlined prescription handling, so that a pharmacy can reduce its opening hours by two hours per day.

7.1.2.Home care

The digital pen ensures that the patient gets the service requested and thus increases security for him or her.Security for elderly in Solna MunicipalityBetter quality of outpatient careAnnual savings of £220,000


23Callaper

7.1.3.Clinical trials

Using the digital pen means no change to work processes, faster access to data and improved accuracy.

Accelerating information flow in clinical trials

Clinical trials world wide aided by digital pen and PaperEnabling broad epidemiological study

7.1.4.Hospital

More natural interaction with patients means better quality of care.Digital Data Capture in the Field of AnaestheticsFull attention given to patientsMore time for care with Digital Pen and Paper

7.2.PUBLIC SECTORAdministration is very time consuming. But there is a cure: digital pen. Some real life examples:

It saves 450 hours of police time every day. In a year, it has freed up caseworker time worth $560,000. Shortening waiting lists and raising productivity

7.2.1.Police

Saving 450 hours of police time every dayBetter management of parking citationsMore time in the field with Digital Pen and Paper

7.3.FIELD SERVICEA digital pen can go with you anywhere. Data can be instantly transmitted to any backend system via a mobile phone. Field workers can do their job the way they always have. No extensive training, no downtime. Some concrete examples:

The work order paper used by a windscreen replacement specialist has been reduced by 80%, corresponding to 8 million pieces of paper annually.

Digital tracking and tracing of consignments has proven to be 90% cheaper than traditional Proof of Delivery systems.

Energy inspectors can now conduct up to ten inspections, instead of only around six per day.


http://www.anoto.com/filearchive/1/14709/pendatasolutions_children_090626.pdf

http://www.anoto.com/filearchive/2/21467/Velosum_100601.pdf

http://www.anoto.com/filearchive/1/16398/WuHu_Longhope.pdf

24Callaper

7.3.1.Inspection

Quicker inspections of life-saving equipmentQuality control made easier with Digital Pen and PaperEasier management of subcontractors

7.3.2.Service and sales

Here are some examples of how the digital pen has improved processes within service & sales.Generating big savings and new revenue for CarmedicDigital Pen and Paper speeds up billingImproving productivity among repair technicians

7.3.3.Transport and logistics

Here are some examples of how the transport and logistics sector has benefited from using the digital pen.

Enabling digital tracking and tracingEnabling aeroplanes to take off on timeEnabling better planned maintenance

7.4.INTERACTIVE LEARNING

7.4.1.Interactive WhiteboardsA regular whiteboard in any material becomes interactive by adding the Anoto surface and an Anoto pen. And this without the need to build in any electronics, since all intelligence lies is in the surface and the pen. Multiple wireless Anoto digital pens can be used in parallel, and the Anoto technology offers the possibility of moving seamlessly between this and other products, such as interactive screens, slates, and paper products. It makes a collaborative learning experience become real.

7.5.PAPER APPLICATIONSA paper product becomes an innovative and interactive educational product by adding the Anoto surface and an Anoto pen. In addition to the great benefit of keeping the intuitive pen and paper experience, the product gets fully interactive. Anoto digital pens may come with microphone, speaker, and display to take your writing experience to an extraordinary level.

Below are some of the reported, remarkable, applications of the digital pen from Anoto.


25Callaper

“The digital pens and the new system have modernised the way we do things…”, says Lynda Findlay, Senior Coordinator, Planning and Control, British Airways.Before a pilot can take off, a rigid procedure is carried out, to ensure that the aeroplane is neither overloaded, nor out of balance. Passengers, baggage and cargo need to be distributed correctly, depending on the aircraft type and the passenger seating arrangements. Previously, the procedure was often time-consuming and a potential cause for delays and missed take-off slots. British Airways has now found a way to speed up the process with a 100% audit trail, near real-time data transmission, and minimal disruption to working practices. The new TRIP (TurnRound Intelligent Pens) solution is based on Anoto’s Digital Pen and Paper technology and is used by over 300 turn-round coordinators to manage all of the airline’s departures from London’s Heathrow and Gatwick Airports. Benefits: Fast, manageable and auditable airside load-control communication during procedures. Prompter departures and a reduction of lost take-off slots.

Annual savings of £220,000

“One of the most interesting benefits that was unplanned in this process is the “Wow!” factor…”, says Nate Beckman, MRM Project Manager, Safelite.

Safelite® AutoGlass is a US-based windscreen replacement specialist. Previously, Safelite dispatched repair technicians and provided administrative reporting manually. Each day, at the beginning of their shift, technicians were required to drive to a central location to pick up work orders and return at the end of their shift to drop off their completed paperwork. Therefore, the only way a technician could be alerted about work order changes was by calls to his Nextel BlackBerry® smartphone. These calls were distracting to the technician, especially if he was in the middle of a job. Moreover, the process required numerous steps to obtain approval prior to beginning the work and to receive final authorization to process the payment. Safelite wanted to create a custom solution that could manage field tasks wirelessly and selected ExpeData® Enterprise Digital Writing platform, based on Anoto Digital Pen and Paper technology for creating its solution. Benefits: Each technician can complete one extra job per day. A 25% reduction in cancellation of jobs. An 80% reduction in work order paper used. Centralized deployment to a diverse population of field technicians. Cost-effective. Playing an important part in disease control

“We are thrilled to see that our solutions have really improved conditions in the region. It is really fantastic“, says Laven Pillay, Managing Director of Xcallibre.


26Callaper

Too little, too late. That has long been the frustrating effect of the United Nations’ drought and infectious disease programs in Sub-Saharan Africa. But all that is history now. Due to a new technical application, information is transmitted in real-time, which means that the UN can actually prevent outbreaks of diseases and the dramatic effects of sudden droughts or floods. This is of key importance, as severe drought since 2000 has lead to limited grazing and water points. Behind the successful turn of events, lies the South African IT-company Xcallibre. The company has developed a unique application based on Digital Pen and Paper technology. With the help of the application, information transfer is accelerated and the users and back offices can simultaneously receive secure and controlled access to the information collected. The solution has also improved the information management capacity, since the data collection methods now can be standardized throughout the region. Benefits: Practically eliminates need for data entry. Easy to use – almost no training required. Provides nearly instant reports to clients.

“The pens are adequately stable and robust for use in everyday rescue and recovery operations. In the event of a large-scale disaster, the time saved on each patient directly benefits others"

Digital pens give the Swiss Army more patient time

Avalanches, flooding, storm damage – when natural disasters and other large-scale calamities result in physical injuries, medical units and emergency services not only have to provide medical treatment, but also logistics, as quickly as possible. Decisions have to be made about admitting the patients at a clinic and about the means of transport for getting them there.

Thanks to a new solution, speed and efficiency is set to rise dramatically for the Swiss Army’s medical corps: TCC Products, a German Anoto partner, has developed a customised software solution based on Anoto Digital Pen and Paper technology – the SanHist Manager.

Benefits: Complete data accuracy, processing of patient data in seconds, and 70% more time for patients

"We have freed up “time to care” equivalent to five fulltime midwives…"

Portsmouth Hospitals NHS Trust has been examining the provision of care in the community. It is usually dispensed to patients at home by midwives and nurses, who record their activities and notes on paper forms. Richard Sargent, ICT Change Control Specialist/Team Leader at the Trust, was on the look-out for technologies that would enable the mid-wives and nurses to take notes and complete paperwork without changing their current working method.

He first came across Anoto Digital Pen and Paper technology in 2008. During a review meeting with the Trust's mobile operator Telefonica O2, he noticed the account manager taking notes using an unusual looking pen. And when a presentation to the team-leaders at the Trust was arranged, the head of the Maternity department immediately liked what she saw.


27Callaper

Benefits: Annual efficiency savings estimated at over £220,000. Time spent on patient administration is halved. Time freed up is equivalent to five full-time midwives. Protection against mislaid, damaged or stolen notes.Improving productivity among repair technicians

8.OUTLOOKThe complexity of Callpapers ranges from mere handwritten ink transmission to blank paper with sophisticated handwriting recognition and document analysis. Accordingly, the next steps are to increase the complexity, and support the full spectrum of Callpapers. More complex documents allow highly sophisticated transactions triggered by just writing on paper. Integrating more powerful handwriting recognition will increase the level of automatization possible, similar to the development in postal automation, which started with printed character recognition and later extended to cursive handwriting recognition.

The calibration method in our current implementation can probably be made more user-friendly. For instance, instead of tapping calibration points, the writer could copy a single calibration mark printed on the paper. Another very appealing solution of the calibration problem is a camera integrated in the sensor clamp attached to the paper, though this requires additional hardware. The camera automatically matches a calibration mark with the specification in the model file whenever the writer attaches the clamp to a designated point on the paper.

A special focus of the future work lies on the generation of model files. Software tools assisting in generating model files for existing documents will allow us to easily enhance today’s documents with Callpaper functionality.


28Callaper

9.REFERENCES [1] http://www.anoto.com.

[2] http://www.siibusinessproducts.com.

[3] S. Jaeger, S. Manke, J. Reichert, and A. Waibel. Online Hand-

writing Recognition: The Npen++ Recognizer. International

Journal on Document Analysis and Recognition, 3(3):169–

180, 2001.

[4] http://www.otmtech.com.

[5] R. Plamondon and S. Srihari. On-Line and Off-Line Hand-

writing Recognition: A Comprehensive Survey. IEEE Trans.

on Pattern Analysis and Machine Intelligence (TPAMI),

22(1):63–84, 2000.


29Callaper

LIST OF FIGURES

Sl No Figure Number Description Page Number

1 3.1. Callpaper Architecture 6

2 5.1 Digital Pen 11

3 5.2 Functional diagram of Digipen 12

4 5.3 Functional Control subsystem 13

5 5.5 Initiating handwriting recording 15

6 5.4 Process operation of Function Control Subsystem 18

7 5.6.A Storing document onto a system 19

8 5.6.B User interface for accessing documents 20

9 6.1 A model file for multiple choice document 21


30Callaper

THE END
