handwriting recognition for a digital whiteboard ... ?· an online handwriting recognition using...

Download Handwriting Recognition for a Digital Whiteboard ... ?· an online handwriting recognition using vector…

Post on 01-May-2019




0 download

Embed Size (px)


Handwriting Recognition for a Digital WhiteboardCollaboration Platform

Lutz Gericke, Matthias Wenzel, Raja Gumienny, Christian Willems, and Christoph MeinelHasso Plattner Institute Potsdam

Prof. Dr. Helmert Str. 2-3, Potsdam, GermanyEmail: {firstname.lastname}@hpi.uni-potsdam.de

AbstractThe research presented in this paper addresseschallenges at the intersection of two disciplines: web based collab-oration using digital whiteboards and handwriting recognition.The main focus is on the handwriting recognition in order toenable asynchronous usage of the whiteboard content beyondthe existing web portal.

We present a way to analyze unstructured whiteboard contentincluding drawings, sketches and handwritten text. Our approachuses a recursive extension of the DBSCAN algorithm in order totransfer smaller portions of content to the recognition engineand achieve an appropriate spatial clustering of the content.The adjustment of the configuration parameters, as well as thedevelopment of a break condition for the recursion, are shown indetail. We show that it is possible to use an online handwritingrecognition engine with offline data and still achieve meaningfulresults. The presented architecture on the one hand, and thecombination of online and offline recognition on the other, easeasynchronous modes of interaction using digital whiteboards.

Keywordsremote collaboration; handwriting recognition;digital whiteboard; DBSCAN


In the beginning, we developed Tele-Board, which is adigital whiteboard system, supporting different modes of workwithin the same tool - asynchronous and synchronous aswell as co-located and distributed settings (cf. [3]). To makeasynchronous working modes more convenient and enablereuse of the produced data, we wanted to find ways of betterdocumenting and computationally understand the content.Based on the data that is already archived in our system,we evaluated different approaches in the field of handwritingrecognition and their applicability for unstructured whiteboardnotes. Applications for the recognized texts are searcheson the whiteboard content or reuse in text-applications, e.g.spreadsheets or presentations.

Handwriting recognition has been a major field of interestfor researchers in machine learning applications. Recently,some very advanced solutions have emerged on the mar-ket. While some solutions use Optical Character Recognition(OCR) to analyze scanned documents, others deal with hand-written text from, e.g., a Tablet PC, usually with the help ofan online handwriting recognition using vector data. Typically,these online handwriting recognition systems have a higherrecognition rate than offline recognition systems [9], since theycan use more features of the writing (e.g., order, pressure ordrawing velocity).

Recognition of handwriting is a complex task [10] andtherefore, results will not be available to the users instantly.Nevertheless, for collaboration tasks, it is often not a goodidea to run an online recognition, because interaction withthe user (e.g., choosing recognition alternatives) would capturetoo much attention. In our application domain - handwritingrecognition on a digital whiteboard system - distractions ofany kind are not welcome since people are focused on thecreative teamwork.

The data we use in the Tele-Board system - which isthe underlying application framework for the developmentdescribed in this article - is vector data. This means thatthe representation of handwritten texts, drawings, etc., is notonly stored as an image but as line strokes consisting ofpoint coordinates. This way, we can benefit from two differentfields: an offline recognition that does not distract usersand a recognition quality as good as an online handwritingrecognition system equipped with more context information.However, we also need an efficient way to analyze the data thathas been archived before the introduction of the handwritingrecognition within the Tele-Board.

Figure 1. Digitally archived image representation of mixed writing/sketchingon a whiteboard

Our approach is an offline handwriting recognition usingan existing online handwriting system with very good resultseven for low-quality writing and different writers. In order torealize this procedure, we have to overcome some problems.The stroke data has to be transferred to the recognition engine

and somehow be replayed there as if it were a live drawing.The problem proves to be even more complex when we gointo the details. Whereas in an online handwriting recognitionsystem the input field is often a small area on the screen wheresingle words or phrases can be written, a whole whiteboard asan input field is a much more complex construct (see figure1). We have to find an automated solution to build clustersof content which can be passed to the recognition engine.Another problem that has to be solved by our presented clus-tering algorithm is the temporal order, which is not necessarilyequivalent to the spatial order of the single strokes.

Towards an understanding of the projects context, we givean architectural overview of the Tele-Board system in thefollowing section. Following, we describe related work in thefield of digital whiteboards, offline and online handwritingrecognition, as well as clustering algorithms. We then show theclustering algorithm and the details of triggering recognition aswell as an evaluation of the recognition results. A conclusionwill summarize this article and give an outlook on furtherresearch activity.


base location A base location B

input devices

workspace hub workspace hub

collaboration server


synchronized design panel

input devices

Figure 2. Tele-Board systems overall setup: two locations are connectedvia a collaboration server, several input devices are used to create content,the whiteboard hardware is used to interact with the whiteboard surface

This section outlines the general architecture of Tele-Board.As shown in figure 2, there are multiple locations (in this caseonly two) that are connected via a communication server. Thisserver synchronizes whiteboard content between the boards. Atouch-sensitive whiteboard input device is used to interact withthe content. Several mobile input devices (iPad/iPod/TabletPC) are used to create content such as handwritten stickynotes or keyboard typed notes from a laptop computer. Thecommunication relies on the open Extensible Messaging andPresence Protocol (XMPP) and whiteboard element data isencoded as XML format. Due to this client server infras-tructure, we are able to capture every single synchronizationcommand between the clients and store the data in a database.This allows us to reconstruct every single point in timeof the whiteboard content history. For more details on thisinfrastructure, see [2].

Whiteboards are organized with the help of projects andpanels. A panel describes one whiteboard session with its

timeline of events (p = (e1, e2, e3, ...)), whereas an eventcan be a NEW, CHANGE or DELETE action on a singlewhiteboard element (path, sticky note, cluster, etc.). A stricttemporal order makes it possible to retrace every elementstimeline and its state at a given time. A project is a set ofpanels (pro = {p1, p2, ...}) used to organize the differentsessions and equip them with a set of permissions [2].4 Struktureller Entwurf der Handschrifterkennungskomponente 31

Abbildung 4.3: PostIt Darstellung auf dem Whiteboard und dessen XML Reprsentation

Attribute, die das Element beschreiben. Die Attribute x und y legen bspw. die Positiondes Mittelpunktes des PostIts auf dem Whiteboard fest. Das Attribut textcontent kanneinen Unicode-Text-String enthalten, der innerhalb des PostIts dargestellt wird, sofernein solcher mittels einer Tastatur angegeben wurde. Der Kindknoten postitpath enthltinnerhalb seines Attributs d einen Zeichenpfad in SVG14 Notation, wobei ausschlie-lich die SVG-Pfad Kommandos moveto (M) und lineto (L) verwendet werden, welcheeine Positionsnderung bzw. das Zeichnen einer Linie bewirken. Dieser Pfad stellt so-mit eine etwaige Schriftreprsentation dar und ist damit die Quelle fr die zuknftigeHandschrifterkennung.

Ein weiteres Whiteboard-Element ist das Skribble. Es handelt sich dabei um einen ein-zelnen Zeichenpfad, welcher direkt auf dem Whiteboard erstellt wird. Das bedeutet, dassein Skribble durch unterbrechungsfreies Zeichnen mittels Stift oder Maus zwischen Auf-setzen und Absetzen des Stifts (bei einer Maus entsprechend das Drcken und Loslassender Maustaste) definiert wird. In Abbildung 4.4 sind die Darstellung auf dem White-board und die Reprsenation durch XML visualisiert. Genau genommen handelt es sichbei der Abbildung auf dem Whiteboard um eine Ansammlung mehrerer Skribbles, dadie Zeichnung und der Text durch mehrfaches Auf- und Absetzen eines Stiftes entstan-den ist. Die XML-Reprsentation eines Skribbles besteht aus einem Knoten mit demAttribut d, das wie bei einem PostIt einen SVG-Zeichenpfad enthlt. Der Unterschieddabei ist allerdings, dass bei einem Skribble immer nur ein M-Kommando am Anfangdes Pfades vorkommt. PostIts enthalten somit alle Zeichenpfade in zusammengefassterForm.

Das Element Cluster dient ausschlielich der Gruppierung und stellt damit eine Mengevon inhaltlich zusammengehrigen Elementen dar, was durch eine Umrandung auf demWhiteboard hervorgehoben wird. Das Cluster-Element in Abbildung 4.5 beinhaltet einPostIt und mehrere Skribbles, welche das Wort Cluster reprsentieren. Ein Cluster14http://www.w3.org/TR/SVG/paths.html

Figure 3. Graphical and XML representation of a sticky note whiteboardelement

For the hand writing recognition (HWR) explained here,a path-based representation of each draw


View more >