A Sketch Recognition System for RecognizingFree-Hand Course of Action Diagrams

T. Hammond, D. Logsdon, B. Paulson, J. Johnston, J. Peschel, A. Wolin, and P. TaeleSketch Recognition Lab, Department of Computer Science and Engineering

Texas A&M University, College Station, TX 77840, srl@tamu.edu

Abstract

Military course-of-action (COA) diagrams are used to depictbattle scenarios and include thousands of unique symbols,complete with additional textual and designator modifiers.We have created a real-time sketch recognition interface thatrecognizes 485 freely-drawn military course-of-action sym-bols. When the variations (not allowable by other systems)are factored in, our system is several orders of magnitudelarger than the next biggest system. On 5,900 hand-drawnsymbols, the system achieves an accuracy of 90% when con-sidering the top 3 interpretations and requiring every aspectof the shape (variations, text, symbol, location, orientation)to be correct.

IntroductionDrawing with pen and paper is a common method for usersto convey visual information. Sketch recognition seeks toharness the power of pen and paper by automatically under-standing freehand-sketched input with a stylus and digitizingscreen. Furthermore, sketch recognition attempts to recog-nize the intent of the user while allowing the user to draw inan unconstrained manner. This permits the user not havingto spend time being trained how to draw on the system, norwill the system need to be trained on how to recognize eachusers particular drawing style.

The recognition of hand drawings has a variety of uses, in-cluding symbol recognition in Computer-Aided Design sys-tems, providing automatic correction or understanding ofdiagrams for immediate feedback in educational settings,functioning as alternative inputs for small keyboard-less de-vices (such as Palm Pilots), or providing gestural interfaces.Our aim as sketch recognition researchers is to build systemsusing artificial intelligence techniques that recognize drawn-diagrams with human-recognizer efficiency and accuracy.

One real-world domain that has thousands of symbols ismilitary course of action (COA) diagrams. Military com-manders use COA diagrams to plan field operations, wherethe symbols represent troops, supplies, obstacles, and move-ment patterns. Currently, commanders draw COAs by handon a map for planning purposes, and then these diagrams areentered into a computer through typical WIMP interfaces forpurposes of simulation and plan communication. Previous

Copyright c© 2010, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

attempts to apply sketch recognition to COAs have resultedin systems that can only recognize a handful of shapes, oftenrequire users to draw each shape with a specific gesture, anddo not allow users to draw freely (Pittman et al. 1996).

In this paper, we present a first-look at an application thatallows military commanders to hand-draw hundreds of COAsymbols directly on a digitized map. The number of symbolswe support is vastly greater than previous sketch recognitioninterfaces, and, by utilizing many artificial intelligence tech-niques, our system can rank the correct interpretation in thetop three returned results 89.9% of the time, for 485 sym-bols.

Implementation and MethodolgyThe flowchart in Figure 1 shows the steps we take to recog-nize drawn course of action symbols. The following sectionsexplain the recognition steps in more detail.

GroupingIn our framework, we have recognizers that work well forshapes and some that work well for text and decision graph-ics. PaleoSketch is used to recognize basic geometric shapeslike rectangles, lines, and ellipses, while our handwritingrecognizer (HWR) is responsible for recognizing text, deci-sion graphics, and echelon modifiers. The job of the group-ing algorithm is to make sure the right strokes go to the rightrecognizer. To perform grouping, we first find the largeststroke present, which is typically the boundary of a unit sym-bol. Next, the remaining strokes are grouped into two sets,depending on whether they are present within or outside ofthe bounding box of the largest stroke (Figure 2). Inside

Figure 2: Example results of the grouping algorithm. The largeststroke is found (red), and the remaining strokes are grouped basedon whether they occur inside (blue) or outside (yellow) of thebounding box of the largest stroke.

Proceedings of the Twenty-Second Innovative Applications of Artificial Intelligence Conference (IAAI-10)

1781

Figure 1: The recognition process, starting from a stroke represented in an XML datafile, to the final, unique symbol identifiers, SIDCs.

strokes are either going to be decision graphics, text, or typemodifiers that need to be sent to the primitive recognizer.These strokes are initially run through the HWR and if thereturned confidence is high enough, they are marked as textor decision graphics. If the confidence is low, then they aresent to PaleoSketch to be recognized as primitive shapes.Strokes that occur outside of the bounding box of the largeststroke are typically echelon modifiers and can recognized bythe HWR.

SegmentationStroke segmentation involves breaking drawn strokes into aset of primitive building blocks, such as lines or arcs. In oursystem, we split every drawn stroke into a series of lines.These polyline representations of the stroke are then sentinto our primitive recognizer, PaleoSketch.

We use a combination of multiple algorithms in or-der to produce a highly accurate polyline segmentation(Wolin, Paulson, and Hammond 2010; Wolin, Eoff, andHammond 2008; Wolin, Paulson, and Hammond 2009;Sezgin, Stahovich, and Davis 2001; Kim and Kim 2006;Douglas and Peucker 1973).

Primitive RecognitionThe goal of primitive recognizers is to classify individualstrokes into one of a set of basic building block shapes.These primitive shapes can then be combined to form morecomplex symbols using high-level shape grammars likeLADDER (Hammond and Davis 2005). For primitive recog-nition, we use an extended version of the PaleoSketch algo-rithm (Paulson and Hammond 2008). This version of Pale-oSketch supports 13 different primitive shapes, which canbe seen in Figure 3.

Figure 3: Primitive shapes supported by our modified version ofPaleoSketch.

Dashed PrimitivesMany symbols in the COA domain are “anticipated” anddrawn with dashed boundaries containing an arbitrary num-ber of lines. To search for dashed boundaries, we start byfirst finding instances of dashed lines. The algorithm startsby iteratively cycling through the shapes on the screen intemporal order and maintaining a stack of lines (and dots)that make up a possible candidate for a dashed line. Aswe encounter conditions that indicate a dashed shape is notpresent (e.g. large slope change, large gap between strokes),the stack is sent to a function that generates a dashed line ifit contains more than one shape. Once all dashed lines havebeen found, they are placed into four different bins basedon their orientation: those that are close to vertical (within15 degrees), close to horizontal, or have a positive or nega-tive slope. Next, we search within the horizontal and verticalbins for possible rectangles, and within the positive and neg-ative bins for diamonds. For dashed ellipses, we perform asimilar operation to that of dashed lines, except that we also

1782

allow for polylines are arcs to be considered as dashes. Oncewe have a stack of candidate dashes for an ellipse, we gener-ate a temporary stroke using the endpoints and midpoints ofthe strokes on the stack. If this generated stroke passes con-ditions of an ellipse test, then we group the strokes togetheras a dashed shape.

Decision Point RecognitionDecision points are typically drawn as 10-line stars in COAdiagrams. Stars are searched for using a similar approach todashed shapes. However, in addition to checking the spatialdistances between consecutive strokes, we also verify thatthe angles between lines alternates between acute and obtuseangles.

Handwriting RecognitionThe handwriting recognizer uses two pre-built multi-layerperceptron models from Weka to recognize inner and eche-lon text, respectively. Echelon text comprises of the symbols‘x’, ‘+’, ‘-’, ‘1’, and ‘*’. Inner text has the character set A-Zand 0-9, with some special symbols used in decision graph-ics (See DecisionGraphics). When strokes are passed to thehandwriting recognizer, they are first grouped to combinethe strokes into logical words and characters based on thestroke’s spatial separation and overlap, and using the left-to-right assumptions of handwriting.

After these stroke groupings are formed, a brute-force ap-proach is taken to compute the character and word possibil-ities. Our domain has no word more than seven letters, andeach character is no more than five strokes, so the numberof possible words is bounded. A confidence value for eachpossible character or word is computed using the multi-layerperceptrons, and the handwriting with the best confidencevalue is used for further symbol recognition.

CALVINCALVIN is a heuristic-driven geometric recognizer basedon LADDER (Hammond and Davis 2005). It builds com-plex high-level shapes by combining shapes (primitves rec-ognized by PaleoSketch (Paulson and Hammond 2008) orother high-level shapes) in different combinations. The waysthat shapes can be combined are dictated by geometric con-straints, such as saying that one shape contains another, orthat the endpoints of two lines are coincident. As an ex-ample, see Figure 4 LEFT, a sketch of a mechanized infantryunit. Some of constraints that might be used to specify howthe parts of this shape fit together include the lines having apositiveSlope and negativeSlopve, as well as theendpoints being coincident with the corners of the rectangle.

In addition to the basic recognition capabilities found inLADDER, CALVIN has been hand-tuned using heuristics toincrease the accuracy and efficiency of recognizing course ofaction diagrams. It re-analyzes the set of shapes for group-ing purposes. Although strokes have already been groupedfor recognition at earlier stages, recognition context can helpus re-group strokes more accurately. For instance, if theprimitive recognizer has detected a rectangle with a highdegree of confidence, then we can say that certain strokes

Figure 4: LEFT: Sketch of a mechanized infantry unit. The con-straints for this symbol dictate that the two diagonal lines thatform the X (the infantry part) are coincident with the re-spective corners of the rectangle, and that one of the lines has apositiveSlope and the other a negativeSlope. The el-lipse (the mechanized part) must be contained within the rectangle,and all the centers must be roughly coincident with each other.RIGHT: Sketch of a mechanized infantry unit with various mod-ifiers. After our recognition algorithms determine there is a rect-angle in the sketch, it can group strokes according to contextualplacement relative to the rectangle. The two Xs above the rectan-gle belong to the unit’s echelon, the minus to the right tells us thatthe unit is reduced, and the alphanumeric identifier at the lower lefthelps us identify this particular unit. Without the context of therecognized rectangle, identifying these other pieces of information(especially differentiating the echelon X from the X in the centerof the rectangle) can become difficult.

Figure 5: Examples of phase lines and boundary lines. On the left,a Phase Line, ‘PL’, is uniquely identified by the identifier, ‘BLUE’.On the top-right, an ambiguous phase or boundary line is drawnhorizontally. On the bottom-right, a Boundary Line is labeled witha company echelon, ‘II’, and the unique identifiers ‘1-2’ and ‘4-5’.

in certain positions relative to that rectangle should go to-gether. After grouping, it is then able to accurately put to-gether the requisite pieces of a military symbol. In Figure 4RIGHT, we show the same sketched unit except with modi-fiers. Once CALVIN determines a rectangle has been drawn,we use contextual rules to determine the inner portions of thesymbol (the X and ellipse), and the different outer portionsof the symbol. In this case, the user has drawn an echelon(the two Xs on top), noted that the unit is reduced (the mi-nus on the right), and assigned an identifier (in the lowerleft) to the unit. Without contextual rules, identifying allthese components simultaneously becomes an NP-completesearch problem.

Phase/Boundary LinesPhase lines and boundary lines define the location steps ofmilitary operations and the boundaries by which militaryunits are confined, respectively (Figure 5). Phase and bound-

1783

Figure 6: On the left, we have a General Belt obstacle symbol.On the right is a Minefield Static obstacle. These obstacles can bedrawn with dynamic patterns, such as with more or less spikes forthe General Belt, or more inner circles in the Minefield Static.

ary lines can be arbitrary contours, and they are recognizedfirst by determining if there is a long stroke of sufficientlength. If a long stroke exists, then the remaining strokesare assumed to be text and are sent to the handwriting recog-nizer. The handwriting recognizer looks for type identifiers,such as ‘PL’ or ‘BL’, and unique identifiers, such as ‘BLUE’or ‘ALPHA’ that could distinguish the phase and boundarylines.

AreasAreas are large, closed strokes that mark zones of interest,such as the landing zones (‘LZ’) shown in the penultimatestep of the flowchart (Figure 1). The area recognizer firstdetermines if the largest stroke in the symbol is closed. If alarge, closed stroke exists, then echelon, type identifier, andunique identifier text is looked for. Echelons are located ateither the top or bottom of the closed stroke, and identifiertext is located inside of enclosure.

Decision GraphicsDecision graphics are type modifiers that include a combina-tion of filled and un-filled rectangles and triangles. Becausethe primitive recognizer is not equipped to handle filled-inshapes, we recognize these symbols using the HWR. Theseshapes are included in the common dictionary that handlesmodifiers that occur within the bounding box of the largestshape on the screen (as determined by the grouper). If theHWR detects instances of decision graphic modifiers, it thenanalyzes all shape definitions that are of the type “Decision-Graphic” to find the correct symbol.

ObstaclesSome course of action shapes are drawn using similar draw-ing patterns, yet can contain a dynamic number of patternrepetitions (Figure 6). These symbols can be recognized bylooking at the symbols on a micro-level. A set of featuresare computed for each symbol, and this “bag of features” iscompared to a set of template patterns. Symbols with similarpatterns can then be recognized.

ArrowsArrows indicate movement of friendly or hostile forces andare allowed to have arbitrary paths. We recognize arrowsby looking at the number of strokes drawn, the number ofline segments the strokes can be broken into, and the type ofarrow head used.

Interface

In producing our COA interface, we have identified threerequirements: quick creation, simplicity, and readability.

First, we must design this interface in such a way thatcourse of action maps can be quickly created. The interfaceis critical because this project is intended to be used in acritical environment under time constraints. Indeed, a majorgoal of this project is to speed up creation of course of actionmaps, so its interface should facilitate this requirement. Allof the recognizers reported above supply confidence values,the interface uses these to order the different results.

Second, we require simplicity. Sketch recognition, bydefinition, aims to simplify the user experience by allow-ing all or most user input in the form of sketching. Thisimplies that our application could be extremely simple, per-haps consisting of only a drawing surface and a handful ofbuttons.

Finally, readability is our third requirement. The sketchrecognition algorithm produces a list of interpretations or-dered by probability. Usually, the desired shape is returnedat the beginning of the list. However, when dealing with alarge set of possible shapes, the desired interpretation mayin fact be second or third in the list. We therefore have todisplay a few top recognition results in a way that keeps theinterface simple and usable. We feel that we have success-fully incorporated these requirements into the interface.

To fulfill our requirements, our interface consists of adrawing panel and a few toolbar buttons. We provide quickcreation and simplicity by including few interface elements,allowing immediate creation upon launching, and automat-ically performing recognition and displaying results. Also,after a shape is drawn by the user, recognition automaticallyoccurs when the hand is removed from the screen. To facili-tate our readability requirement, the top results are displayedin panels once recognition finishes. These panels displaytextual information about the possible symbols as well asa preview image. We have displayed this information in amanner that is easily readable. While these core require-ments have been implemented, we have added a few morefeatures to enhance the creation process.

We have included a few intelligent features that help im-prove the user experience. First, we have included strokebeautification (Figure 7). This occurs on two levels. Inmany symbols, a user’s strokes are replaced by a correctsymbol image file. Some symbols, however, are definedby user-drawn boundaries and, therefore, do not have im-age files associated with them. Many of these amorphousshapes contain some text labels which are written in by theuser. Our interface uses handwriting recognition to converthandwriting into text while keeping the user-defined bound-ary in these types of shapes. In addition to stroke beautifi-cation, we have included a set of basic editing tools includ-ing move, duplicate, and delete. These tools, while simple,provide some much needed functionality. We have also im-plemented a fourth tool to convert a beautified symbol to itsuser-drawn strokes and back again. This obviously helps usin testing our application, but it also can help users under-stand the sketch recognition process.

1784

(a) The user drew COA symbols in our interface using a stylus. (b) Drawn symbols are beautified, where some of the symbols arereplaced with their recognized images, and the text becomes typed.

Figure 7: Our interface incorporates stroke beautification, which takes recognized strokes (7(a)) and turns them into cleaned images (7(b)).

Results and DiscussionWe trained our recognizers on a set of 485 different sym-bols, collecting over 6,000 samples drawn by researchersand course of action diagram experts. We tested on a dif-ferent set of data consisting of 5,900 samples drawn by 8 re-searchers (not that none of these users were included in thedata used for the development and training of the system).

We return multiple interpretations for each symbol recog-nized, since many of the symbols are often similar with onlyminor nuances. The correct interpretation is returned as thetop result 84.4% of the time, in the top three results 89.9% ofthe time, and in the top ten results 90.8% of the time. Theserecognition results are on par with current sketch recognitionsystems, and they are phenomenal when considering that wehave an order of magnitude more symbols than other appli-cations support.

We have not conducted large-scale user interface eval-uations at this time or deployed our interface with mil-itary experts, but we hope to do so in future re-search. A video of the current system is posted here:http://srlweb.cs.tamu.edu/srlng/research/project/22

Related and Prior WorkFrameworks and systems have been created that either canspecified to handle sketched course of action shapes, or pro-vide alternative implementations in doing so. We discusssuch works and discuss their limitations that have been im-proved upon by our implementation.

QuickSet SystemAn early system that was used for battle plans involvingcourse of action shapes was QuickSet (Cohen et al. 1997), a

multimodal system utilizing a multi-agent architecture thatcan be applied for a number of applications. One suchapplication was an interface for inputting course of actionshapes was LeatherNet, an interface that requires users touse a combination of speech and sketch input for drawingthe shapes. The multimodal input activates an over-the-shelf speech recognizer, as well as a traditional pen-basedgesture recognizer consisting of a neural network and hid-den Markov models, whenever the user places the pen onscreen. The system therefore listens to both speech and ges-ture simultaneously during interface usage. While Quick-Set demonstrated that it can complete the task faster thanexisting CAD-style interfaces at the time (Forbus, Usher,and Chapman 2003), the necessity of speech as a modalityfor users using the system to describe the layout of forcesin simulated exercises using course of action shapes haveexhibited shortcomings. Specifically, even current speechsystems are severely limited in noisy environments, and tar-get users of such a system have repeated reported that theywould not use it if it has a speech recognition requirement(Forbus, Usher, and Chapman 2003), which is the case forthe QuickSet system.

nuSketch Battlespace InterfaceThe nuSketch Battle Interface (Forbus, Usher, and Chap-man 2003) is a more recent system for handling sketchedcourse of action shapes, and was designed to address theshortcomings of systems such as QuickSet. The first no-table contrast of the nuSketch approach to previous systemsis its focus on sketching and the lack of a speech modal-ity. Despite the systems heavy emphasis on sketching fordrawing up battle plans, the sketch-based interface does notutilize recognition on the users input. Instead, the system

1785

relies on visual and conceptual understanding of the input todrive the drawing up of battle plans using sketched courseof action shapes, and relies more on reasoning techniques ofthe sketched shapes themselves. In order to address the lackof recognition techniques, the system also depends heavilyon interface mainstays such as toolbars, combo boxes, andtype-in boxes to facilitate inputting the sketches.

LADDER framework

One framework that has served as the basis to our work isthe LADDER framework (Hammond and Davis 2005), adomain-independent sketching language that can be used todescribe how shapes and shape groups are drawn, edited,and displayed in the recognition process. The language isused as a tool to describe higher level symbols as a combi-nation of lower-level primitives that fulfill certain geometricconstraints, and is useful for recognizing symbols in a do-main absent of domain-specific information. Examples ofdomains that have successfully utilized LADDER includebiology, music notation, East Asian characters and sketch-based educational games. A limitation of LADDER thoughis that it does not handle symbols that are difficult to de-scribe geometrically or are ambiguous without context, as isthe case with many course of action shapes.

COA Design Interface

The COA Design Interface is a system that addresses thelimitations of LADDER for sketched course of action shapesby extending that frameworks capabilities (Stolt 2007). Be-sides modifying LADDER to make it domain-specific tosketched course of action shapes, one significant extensionis the use of the Intermediate Feature Recognizer (IFR). Thisrecognizer serves multiple purposes such as recognizing cer-tain shapes (e.g., variably-drawn) not handled by LADDERwell, as well as provide error correction. The IFR allows alarger variety of shapes to be handled with reasonable ac-curacy, and thus is capable of reporting up to 247 shapes.The current implementation is still lacking components thatare critical to military planners drawing battle plans, such ascourse of action shapes that contain text.

ConclusionThis paper describes a sketch recognition system that recog-nizes military course of action diagrams. The sketch recog-nition system recognizes 485 different military course-of-action diagram symbols, with each shape containing its ownelaborate set of text labels and other variations. Even with-out the variations and text this is well over an order of magni-tude more symbols than the next largest system. When onefactors in the variations (not allowable by other systems),this system is several orders of magnitude larger than thenext biggest system. On 5,900 student hand-drawn symbols,the system achieves an accuracy of 90-percent when consid-ering the top 3 interpretations and require every aspect ofthe shape (variations, text, symbol, location, orientation) tobe correct.

AcknowledgmentsThe authors would like to acknowledge Christine Alvarado,Paul Corey, Daniel Dixon, Brian Eoff, Marty Field, RobertGraham, Brandon Kaster, Walter Moriera, Manoj Prasad,Pat Robinson, Cassandra Rutherford, Metin Sezgin, andYuxiang Zhu, in their participation in and discussion of theideas of this work. This work is supports in part by NSFgrants 0757557 and 0744150.

ReferencesCohen, P. R.; Johnston, M.; McGee, D.; Oviatt, S.; Pittman, J.;Smith, I.; Chen, L.; and Clow, J. 1997. Quickset: multimodalinteraction for simulation set-up and control. In Proceedings of thefifth conference on Applied natural language processing, 20–24.Morristown, NJ, USA: Association for Computational Linguistics.Douglas, D., and Peucker, T. 1973. Algorithms for the reduction ofthe number of points required to represent a digitized line or its car-icature. Cartographica: The International Journal for GeographicInformation and Geovisualization 10(2):112–122.Forbus, K. D.; Usher, J.; and Chapman, V. 2003. Sketching formilitary courses of action diagrams. In IUI ’03: Proceedings of the8th international conference on Intelligent user interfaces, 61–68.New York, NY, USA: ACM.Hammond, T., and Davis, R. 2005. LADDER, a sketchinglanguage for user interface developers. Computers & Graphics29(4):518–532.Kim, D., and Kim, M.-J. 2006. A curvature estimation for peninput segmentation in sketch-based modeling. In Computer-AidedDesign, 238–248.Paulson, B., and Hammond, T. 2008. PaleoSketch: Accurate prim-itive sketch recognition and beautification. In Proceedings of the13th International Conference on Intelligent User Interfaces (IUI),1–10. New York, NY, USA: ACM.Pittman, J. A.; Smith, I. A.; Cohen, P. R.; Oviatt, S. L.; and Yang,T. C. 1996. Quickset: A multimodal interface for military sim-ulation. In Sixth Conference on Computer-Generated Forces andBehavioral Representation.Sezgin, T. M.; Stahovich, T.; and Davis, R. 2001. Sketch basedinterfaces: Early processing for sketch understanding. Workshopon Perceptive User Interfaces.Stolt, K. 2007. Sketch recognition for course of action dia-grams. Master’s thesis, Massachusetts Institute of Technology,Cambridge, MA.Wolin, A.; Eoff, B.; and Hammond, T. 2008. Shortstraw: A simpleand effective corner finder for polylines. In Eurographics 5th An-nual Workshop on Sketch-Based Interfaces and Modeling, 33–40.Wolin, A.; Paulson, B.; and Hammond, T. 2009. Sort, merge, re-peat: an algorithm for effectively finding corners in hand-sketchedstrokes. In SBIM ’09: Proceedings of the 6th Eurographics Sympo-sium on Sketch-Based Interfaces and Modeling, 93–99. New York,NY, USA: ACM.Wolin, A.; Paulson, B.; and Hammond, T. 2010. Combining cor-ners from multiple segmenters. In Submission.

1786