use of adaptive segmentation in handwritten phrase recognition

*Corresponding author. Tel.: #1-716-645-6164-103; fax:#1-716-645-6176.E-mail address: [email protected]!alo.edu (V. Govindaraju).

Pattern Recognition 35 (2002) 245}252

Use of adaptive segmentation in handwritten phraserecognition

Jaehwa Park, Venu Govindaraju*

Center of Excellence for Document Analysis and Recognition (CEDAR), Department of Computer Science and Engineering,University at Buwalo, Buwalo, Amherst, NY 14260, USA

Received 11 August 2000; accepted 14 November 2000

Abstract

Research in handwriting recognition has thus far been primarily focused on recognizing words and phrases. In fact,phrases are usually treated as a concatenation of the constituent words making it in essence an enhanced wordrecognizer. In this paper we present a methodology that will take advantage of the spacing between the words in a phraseto aid the recognition process. The novelty of our approach lies in the fact that the determination of word breaks is madein a manner that adapts to the writing style of the individual. The parameters that decide whether a particular gapbetween components is an inter-word gap or an inter-character gap are computed without the necessity of generalizingover a large training set. Rather, it is tuned to the distribution of the gaps within the instance of the phrase image beingexamined. We compare our approach to the methods described in the literature that simply ignore the signi"cance ofgaps in a phrase. Our experiments show an improvement of about 5% in recognition rates. On a test set of about 1400phrase images the segmentation method `missesa only 2% of the true word break points. � 2001 Pattern RecognitionSociety. Published by Elsevier Science Ltd. All rights reserved.

Keywords: Phrase recognition; Segmentation; Word gaps; Distance metric; Dynamic programming

1. Introduction

Most architectures of handwritten word recognitionsystems described in the literature organize the stages ofa phrase recognizer serially [1]. The preprocessing stageis primarily related to image-processing operations (suchas normalization) that remove irregularities in hand-writing. The segmentation stage divides an image intomeaningful units for recognition. The recognition stage isabout classi"cation algorithms using a lexicon. Given theimage of a handwritten word and a lexicon of possiblewords, the task is one of ranking the lexicon based on the`goodnessa of match between each lexicon entry and theword image. Typically, the word recognizer computes

a measure of `similaritya between each lexicon entry andthe word image and uses this measure to sort the lexiconin descending order of the similarity measure.Recognition of handwritten characters and words has

been extensively studied in the literature [2}4]. Some ofthese methods have been successfully extended to phraserecognition in applications such as postal address inter-pretation [5], bank check processing [6] and text formprocessing [7]. Phrase recognition methods extend theword recognition methods by simply ignoring the gapsbetween the words during the lexicon generation process(Fig. 5). Thus, in e!ect, the phrase recognizers behave likeenhanced word recognizers. This paper addresses thisvery issue. We believe that there is a wealth of informa-tion in the spacing between words in a phrase and shouldnot be ignored. Thus, "nding the gaps between the com-ponents of a phrase and classifying them as either a trueword gap or simply an inter-character gap becomes animportant task. To this end, we have developed a noveltechnique that makes this classi"cation by adapting

0031-3203/01/$20.00 � 2001 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.PII: S 0 0 3 1 - 3 2 0 3 ( 0 0 ) 0 0 1 7 6 - X

Fig. 2. Inter-component gaps using geometric relationship between components generated by (a) bounding box method, (b) runlength-based method, and (c) convex hull-based method.

Fig. 1. Street name recognition in postal applications poses theproblem of recognizing phrases. Note how there are gaps withina word (inter-character gap) and between words (inter-word gap).The problem is one of measuring the gaps between adjacentcomponents and classifying them as inter-word and inter-charac-ter gaps.

parameters to the writing style present in the phraseimage being examined.The algorithm we describe to segment a phrase into

words is tuned to err on the side of over-segmentationrather than under-segmentation. In other words, the trueword breaks in a phrase must be always picked. In theprocess, it is possible that certain words are broken up aswell. However, two adjacent words in a phrase are neverleft joint after the segmentation process. Our methodachieves perfect segmentation in 48% of the cases and`missesa the segmentation points in only 2% of the cases.The remaining fall into the over-segmentation category.We present a matching algorithm that takes as input

the word segments derived by the segmentation algo-rithm and matches them with substrings of lexicon en-tries (called lexemes). Compared to the method in Ref. [3]our approach achieves an improvement of about 5%.This paper is organized as follows. Section 2 summar-

izes previous approaches of word segmentation. Section3 presents the adaptive word segmentation algorithm.Section 4 describes a dynamic phrase-recognition methodthat matches the image segments against lexemes. Section5 presents experimental results and a comparison withmethods that treat a phrase as a single concatenatedword. Section 6 makes some concluding remarks.

2. Previous work

Word segmentation in handwriting is non-trivial giventhat words tend to #ow together.Methods that search for

the likely boundaries of words are somewhat tied to theword recognition algorithms employed and there area number of tradeo!s to be considered [8,9]. Thesemethods assume that gaps between words are larger thanthe gaps between characters. However, exceptions to thisassumption are frequent because of the presence of #our-ish in writing styles with leading and trailing ligatures.Fig. 1 illustrates the di$culty of the task in postal ap-plication of reading street names.There are a limited number of research reports that

present a comprehensive discussion on this topic[6,8}16].Word separation method using spatial distance cues

consists of the following steps: (i) determine the connec-ted components in the given line, (ii) compute the dis-tance (or gap) between pairs of adjacent components, (iii)sort the gaps in descending order of magnitude, and (iv)classify the gaps into inter-word gaps and inter-charactergaps by choosing a threshold. Gaps greater than thethreshold are deemed to be word separation points.Computing the distance between adjacent components

((ii) above) is a challenging research issue. The objective isto obtain an estimate of the inter-component gaps asperceived by humans. The "rst simple-minded estimationmethod computes the horizontal distance between thebounding boxes of adjacent components, where thebounding box of a component is de"ned as the smallestrectangle enclosing the component. The second method[9] uses run-lengths and Euclidean distances between con-nected components and heuristics. Heuristics are used tohandle cases where adjacent components do not havesu$cient overlap in the x and y directions. The thirdtechnique approximates the gaps between componentsby the distance between their convex hulls [8]. Fig. 2illustrates the gaps estimated by these techniques be-tween two connected components. The bounding boxmethod fails to report any gap in the example (Fig. 2a).The convex hull method reports a reasonable estimate ofthe gap in the example (Fig. 2c).Kim et al. describe an intelligent word segmentation

method that incorporates cues that humans use and doesnot rely solely on the one-dimensional distance betweencomponents [17]. The author's writing style in terms ofspacing is captured by characterizing the variation ofspacing between adjacent characters as a function of thecorresponding characters themselves. The notion of ex-pecting greater space between characters with leading

246 J. Park, V. Govindaraju / Pattern Recognition 35 (2002) 245}252

Fig. 3. Stages in word segmentation of a phrase image: (a) slantand skew normalized contour, (b) segmented primitives,(c) stroke interval estimation to derive the prime frequency,(d) generation of two word image segments.

and trailing ligatures is encoded into the segmentationscheme. However, this method, while addressing the indi-vidual writing styles, is still reliant on a training a neuralnetwork for the large sets of data collected across manydi!erent writers. Our approach will primarily addressthis issue in the next section.

3. Adaptive word segmentation

This section describes our approach to "nding theinter-word gaps in a phrase. There are two salient aspectsto our approach: (i) it is not dependent on a large set oftraining data collected across a variety of writers, and (ii)the metric measuring the magnitude of gaps betweencomponents is statistically computed using the featuresof the phrase image under examination.Unlike machine-printed text where the spacing be-

tween characters and words is uniform, in handwritinga writer's inter-character spacing style is quasi-regulatedby the distance between the adjacent vertical strokes.Spaces in a phrase can be between characters (inter-character) and between words (inter-word) (Fig. 1).Images are "rst normalized for slant, skew(slope), aspect

ratio, stroke width, size and rotation [4,18,19]. Piecewiselinearized contour model is used for extracting features forslant and skew detection. High-curvature points on con-tours are extracted and contour segments between thesepoints are assumed to be linear. Vertical and near-verticallines are extracted by tracing the critical points. The globalslant angle is estimated by averaging all the angles of thevertical lines, weighted by their length.Two reference lines, halyine and baseline are used to

establish the horizontal primary writing band. Thecenterline is a line parallel to the x-axis passing the globalmass center of all contours. The baseline is the best "ttingline of local minimum points of concavities of exteriorcontours below the center line (extracted from normaliz-ation step). The hal#ine is parallel to the baseline and isthe best "t to local maximum points of convexities ofexterior contours above the center line. These lines areextracted by angular histograms.Connected components are split into symbolic primi-

tives using convexity and concavity of ligatures.Fig. 3 shows the steps in stroke segmentation. A strokeprimitive is classi"ed as prime or auxiliary. Prime primi-tives are basic primitives where most of the characterbody is in the primary writing band. Auxiliary primitivesare "rst classi"ed as Hats, Tails, and Ligatures. Hat primi-tives are located above the primary writing band (e.g., &T'and i-dots). Tail primitives are located below the primarywriting band (e.g., descender in &y' and comma). Ligatureprimitives are strokes between characters or horizontalstrokes in cursive characters such as &w' and &u'.We make the following assumptions: (i) a character

consists of one or more primitives and at least one prime

primitive, and (ii) the distance between two adjacentprime primitives of a character is equal or less than thedominant prime primitive frequency.After determining the type of each primitive, the aver-

age interval of prime stoke is determined. All primitivesare sorted in the order of x coordinate of mass center ofits contours from left to right. The base sample set is thearray of prime primitives. In Fig. 3c, prime primitives areshown in bounding boxes. The vertical cross lines andprojected points to baseline of mass center are alsoshown.We de"ne the interval of a prime primitive as the

horizontal distance between the center points of the cur-rent and succeeding prime primitives. Let us assumeM primitives are generated in a given phrase image andN primitives are classi"ed as prime. Let i be index ofprime primitives (in 0&N!2) and (x

�, y

�) denote the

center point of mass of ith prime primitive.The interval of the ith prime primitive d

�is given by

d�"x

��!x

�. (1)

The average of intervals dM and variance v� are given as

dM "1

N!1

��

d�, (2)

v�"

1

N!1

��

(d�!dM )�, (3)

where dM is an average of inter-stroke, inter-character andinter-word intervals. dM cannot be a period of prime fre-quency. Assuming that inter-word gap is larger than theinter-character gap and the number of inter-word gaps issigni"cantly smaller than inter-character or inter-stroke

J. Park, V. Govindaraju / Pattern Recognition 35 (2002) 245}252 247

Fig. 4. The bounds of number of characters in each of the imagesegments of the phrase image shown are estimated.

Fig. 5. Original lexicon to be matched against the phrase imageis shown in (a). All the variants of the lexicon entries that areequivalent are shown in (b).

gaps, the prime interval samples can be re"ned by usingdM and v.Let ¹(d) be a "ltering function as follows:

¹(d)"�d dM !�v)d)dM #�v,

0 otherwise.(4)

The prime period p is de"ned by the average value of"ltered prime intervals as

p"

1

K

��

¹(d�), (5)

where K is the number of "ltered prime intervals. Sincehandwriting does not guarantee uniform spacing amongstokes or characters, an error rate e is de"ned by thevariance of "ltered prime intervals that support the con"-dence of the extracted prime period.

e�"

1

K

��

(¹(d�)!p)�. (6)

Finally, a metric of word gap con"dence c�in the ith

prime primitive is de"ned in terms of the MahalanobisDistance as

c�"�

0, d�)p,

d�!p

eotherwise.

(7)

After computation of all word gaps and their con"den-ces, word break points are detected by simple thre-sholding (Fig. 3(d)).

4. Phrase recognition

We describe in this section, a phrase recognizerthat maximally utilizes the information about inter-word and inter-character gaps. A dynamic matchingalgorithm is used which considers combinations of wordimage segments and matches them against appropriateparts of lexicon entries (lexemes). To improve e$ciency,each word image segment matches only a subset of thecharacters in the lexicon entry based on an estimate ofthe number of characters in the image segment derivedfrom the prime frequency. A set of hypotheses are con-structed for potential matches between image segmentsand lexicon entries. These are submitted to a word recog-nizer.The number of characters within an image segment is

estimated by counting the number of times the distancebetween the prime primitives (Fig. 4) is larger than theprime frequency, while skipping prime primitives whosedistance from the previous one (scanning left to right) isless than or equal to the prime period.

Because of #ourishing and non-uniform character sizeor spacing present in handwriting, the accuracy of theestimate of number of characters strongly depends onthe writing style. An error range is de"ned to support theestimate. Letm be a total number of prime primitives andn be an estimated number of characters. The error ranger is given by

r"ne

p. (8)


Fig. 6. Each image segment can potentially match with different substrings of the lexion entries (lexemes). The tables above show how weuse the estimate of the number of characters in each image segment to generate the hypothesis for matching image segments againstvarious lexemes.

Fig. 7. Performance of the phrase recognition method describedusing the adaptive segmentation algorithm.

The boundary of estimated number of characters isgiven by

n��

"�n#�r if n#�r(m,

m otherwise,(9)

n��

"�n!�r n!�r'1,

1 otherwise,(10)

where � and � are control parameters.Using the estimate of the number of characters in an

image segment (hypothesized to be a word) the followingcommon sense conditions hold. This screening processallows us to con"ne the search of the dynamic matchingalgorithm.

1. The number of words in a lexemes should be smallerthan the number of image segments.

2. The number of characters in a lexemes is equal to orless than the sum of n

��over all the segments being

compared.3. The number of characters in a lexemes is equal to or

greater than the sum of n��

over all the segmentsbeing compared.

4. If the number of words in the lexemes is the same asthe number of segments, then the number of charac-ters in each segment is within the bounds set byn��

and n��

.

The word segmentation method segments the phraseimage (Fig. 4). There are "ve image segments identi"ed.The "rst three segments are marked as `de"nitea and the


Fig. 8. Performance of the segmentation algorithm.

Fig. 9. A comparison between the two methods of word mode and phrase mode of recognition at various operating points of error.

remaining two are identi"ed as `possiblea words. Thisimplies that the last two word segments can either beconsidered as individual words or be considered asa word when combined with the third segment.The phrase image has some extraneous components

(`2nd Fla) which are not anticipated in the lexicon (Fig.4a). The lexicon entries of Fig. 4a are expanded to coverall equivalent variants as shown in Fig. 4b.Fig. 4 shows an estimate of the number of characters

we expect to see in each image segment. This estimateallows us to choose the substring from each lexicon entryof Fig. 5. Fig. 5a shows the lexicon substring (lexemes)and the image segment(s) that it can potentially match.Fig. 6b shows how a lexicon entry of Figurehypa can

be constructed using Fig. 6a. � in the bale representsa null matching to account for extraneous components.For example, `W(5) Putnam(6) Ave(7)a of Fig. 6b match-es `Wa with the image segments shown on row index 5 ofFig. 6a; `Putnama matches with the image segmentsshown on row index of Fig. 6a; `Avea matches with theimage segments of row index of Fig. 6a. This allows us to

construct the appropriate lexicon for each image segmentthat can be then submitted to a word recognizer. Eachpotential match is considered a hypothesis.Because hypotheses consist of subset of images and

possible lexicons, a lexicon-driven word recognizer ispreferred for hypotheses veri"cation rather than charac-ter-based recognizer [3].

5. Experiments

The phrase recognition methodology presented in thispaper is applied to street name phrases. A set of 1342images is used as the testing set. The "rst 150 images areused for tuning the parameters. Fig. 7 shows the phraserecognition performance. The word segmentation algo-rithm `missesa the actual word segmentation points inabout 2% of the images while achieving perfect wordsegmentation in 48% cases and over-segmenting in 30% ofthe images (Fig. 8). Figs. 9 and 10 show the performance atvarious operating points of error rates and rejection rates.


Fig. 10. A comparison between the two methods of word mode and phrase mode of recognition at various operating points of rejection.

6. Conclusion

Research in handwriting recognition has thus far beenprimarily focused on recognizing words and phrases. Infact, phrases are usually treated as a concatenation of theconstituent words making it in essence an enhanced wordrecognizer. We have presented a methodology that willtake advantage of the spacing between the words ina phrase to aid the recognition process. The novelty ofour approach lies in the fact that the determination ofword breaks is made in a manner that adapts to thewriting style of the individual. The parameters that deci-de whether a particular gap between components is aninter-word gap or an inter-character gap are computedwithout the necessity of generalizing over a large trainingset. Rather, it is tuned to the distribution of the gapswithin the instance of the phrase image being examined.We have compared our approach to the methods de-scribed in the literature that simply ignore the signi"-cance of gaps in a phrase. Our experiments show animprovement of about 5% in recognition rates. On a testset of about 1400 phrase images the segmentationmethod `missesa only 2% of the true word break points.

References

[1] G. Kim, V. Govindaraju, S. Srihari, Architecture for hand-written text recognition systems, Int. J. Document Anal.Recognition 2 (1) (1999) 37}44.

[2] J.T. Favata, G. Srikantan, A multiple feature/resolutionapproach to handprinted digit and character recognition,Int. J. Imaging Systems Technol. 7 (1996) 304}311.

[3] G. Kim, V. Govindaraju, A lexicon driven approach tohandwritten word recognition for real-time applications,IEEE Trans. Pattern Anal. Mach. Intell. 19 (4) (1997)366}379.

[4] A.W. Senior, A.J. Robinson, An o!-line cursive hand-writing recognition system, IEEE Trans. Pattern Anal.Mach. Intell. 20 (3) (1998) 309}321.

[5] E. Cohen, J. Hull, S.N. Srihari, Control structure for inter-preting handwritten address, IEEE Trans. Pattern Anal.Mach. Intell. 16 (10) (1994) 1049}1055.

[6] J.C. Simon, O. Barat, N. Gorski, A system for the recogni-tion of handwritten literal amounts of checks, Inter-national Conference on Document Analysis System,Kaiserslauterm, Germany, 1994, pp. 135}155.

[7] S.N. Srihari, Y.C. Shin, V. Ramanaprasad, D.S. Lee, Nameand address block reader, Proc. IEEE 84 (7) (1996)1038}1049.

[8] U. Mahadevan, R.C. Nagabushnam, Gap matrices forword separation in handwritten lines, International Con-ference on Document Analysis and Recognition, Mon-treal, Canada, 1995, pp. 124}127.

[9] G. Seni, E. Cohen, External word segmentation of o!-linehandwritten lines, Pattern Recognition 27 (1) (1994) 44}52.

[10] H. Baird, K. Thompson, Reading chess, IEEE Trans. Pat-tern Anal. Mach. Intell. (1990) 552}559.

[11] R.O. Duda, P.E. Hart, Experiments in recognition ofhand-printed text, AFIPS Conference Proceedings, 1968,pp. 1139}1149.

[12] E. Cohen, J. Hull, S. Srihari, Understanding handwrittentext in a structured environment: determining zip codes in


addresses, Int. J. Pattern Recognition Artif. Intell. (1991)221}264.

[13] M. Brady, Toward a computational theory of early visualprocessing in reading, Visible Language 15 (2) (1981)138}215.

[14] M.D. Garris, Unconstrained handprint recognitionusing a limited lexicon, Proceedings of the SPIE Sympo-sium on Electronic Imaging Science and Technology(Document Recognition), San Jose, CA, Vol. 2181, 1994,pp. 36}46.

[15] A.M. Gillies, M.J. Ganzberger, R.M. Rovner, D.J. Hepp,Matching database record to handwritten text, Proceed-ings of the SPIE Symposium on Electronic Imaging

Science and Technology (Document Recognition), SanJose, CA, Vol. 2181, 1994, pp. 66}75.

[16] K. Han, I.K. Sethi, O!-line cursive writing segmentationscheme, Proceedings of the ICDAR, October 1995,pp. 894}897.

[17] G. Kim, V. Govindaraju, Handwritten phrase recognitionas applied to street name images, J. Pattern Recognition 31(1) (1998) 41}51.

[18] R.M. Bozinovic, S.N. Srihari, O!-line cursive script wordrecognition, IEEE Trans. Pattern Anal. Mach. Intell. 11 (1)(1989) 68}83.

[19] J.C. Simon, O!-line cursive word recognition, Proc. IEEE80 (7) (1992) 1150}1161.

About the Author*JAEHWAPARK received the B.S. and the M.S. degree in Electronic Engineering from Hanyang University, Seoul,Korea, in 1989 and 1991, respectively. He received the Ph.D. degree in Electrical Engineering from the State University of New York(SUNY) at Bu!alo in 2000. He is currently a research scientist of the Center for Document Analysis and Recognition (CEDAR) atSUNY at Bu!alo. His research interests include pattern recognition, computer vision and digital signal processing.

About the Author*VENUGOVINDARAJU received his Ph.D. in Computer Science from the State University of New York at Bu!aloin 1992. He has co-authored more than 100 technical papers in various international journals and conferences and has one US patent.He is currently the associate director of CEDAR and concurrently holds the associate professorship in the department of ComputerScience and Engineering, University at Bu!alo. He is the associate editor of the Journal of Pattern Recognition and the area chair of theIEEE SMC technical committee for pattern recognition. Dr. Govindaraju has been a co-principal investigator on several federallysponsored and industry-sponsored projects. He is presently leading multiple projects on postal applications. He is the ProgramCo-chairof the upcoming International Workshop on Frontiers in Handwriting Recognition in 2002. He is a senior member of the IEEE.


use of adaptive segmentation in handwritten phrase recognition

Documents