Versatile Search of Scanned Arabic Handwriting

Sargur N. Srihari, Gregory R. Ball and Harish SrinivasanCenter of Excellence for Document Analysis and Recognition (CEDAR)

University at Buffalo, State University of New YorkAmherst, New York 14228

{srihari, grball, hs32}@cedar.buffalo.edu

AbstractSearching scanned handwritten documents is a rel-

atively unexplored frontier for documents in anylanguage. In the general search literature retrievalmethods are described as being either image-basedor text-based with the corresponding algorithms be-ing quite different. Versatile search is defined asa framework where the query can be either a tex-tual string or an image snippet in any language andthe retrieval method is a fusion of text- and image-retrieval methods. An end-to-end versatile systemknown as CEDARABIC is described for searchinga repository of scanned handwritten Arabic docu-ments; in addition to being a search engine it in-cludes several tools for image processing such as lineremoval, line segmentation, creating ground-truth,etc. In the search process of CEDARABIC the querycan be either in English or Arabic. A UNICODEand an image query are maintained throughout thesearch, with the results being combined by an arti-ficial neural network. The combination results arebetter than each approach alone. The results can befurther improved by refining the component pieces ofthe framework (text transcription and image search).

1 Introduction

While searching electronic text is now a ubiquitousoperation, the searching of scanned printed docu-ments such as books is just beginning to emerge.The searching of scanned handwritten and mixeddocuments is a virtually unexplored area.

Processing handwritten Arabic language docu-ments is of much current interest. One unsolvedproblem is a reliable method, given some query, tosearch for a subset among the many such documents,similarly to searching printed documents. The prob-lem is challenging because of the unique structuralfeatures of Arabic script and the relative infancy ofthe field of handwriting processing.

Content-based information retrieval (CBIR) is abroad topic in information retrieval and data mining

[1]. CBIR algorithms are quite different for the tasksof text retrieval and image retrieval. Correspond-ingly there are two approaches to searching scanneddocuments, stemming from the two different schoolsof thought. One approach is to use direct contentbased image retrieval (word spotting). Another is toconvert the document to an electronic textual rep-resentation (ASCII for English and UNICODE forArabic) and search it with text information retrievalmethods used routinely with electronic documents.Both of these approaches can be successful underideal circumstances, but such a situation is difficultto achieve with current technology of handwritingrecognition. Image based searches do not always re-turn correct results. Arabic handwriting recognitiontechnology does not come close to allowing full tran-scriptions of unconstrained documents. However, bycombining these two methods together, we achievebetter performance than either on its own.

The paper describes a framework for versatilesearch of Arabic handwritten documents. By ver-satile search, we mean both versatility in the queryand versatility in the search strategy–combining con-tent based image retrieval and text-based informa-tion retrieval. Versatality in the query refers to tothe query being either in textual form or electronicform. Another characteristic of versatile search isthat the query can be in multiple languages such asEnglish and Arabic. In the versatile search processboth the original scanned image and the (partial)transcription are maintained at all stages. Searchesproceed in parallel on both document representa-tions. Any query is also split into both an imageand a UNICODE representation which act on thecorresponding instance of the document. The re-sults from both parallel searches are combined intoa single ranking of candidate documents.

The rest of the paper is organized as follows. Sec-tion 2 describes previous work in scanned documentretrieval. Section 3 describes the nature of queriesfor versatile search. Section 4 describes the overall

system architecture as well as the process of segmen-tation necessary for image indexing and retrieval.Section 5 describes the image-based word shapematching approach. Section 6 describes the text-based approach that is based on character recogni-tion. Results in the form of precision-recall with acorpus of handwritten Arabic are described in Sec-tion 7. Conclusions and future directons are givenin Section 8.

2 Related Work

Searching scanned printed documents in English hashad significant success. Taghva et al showed [13]that information retrieval performance continues tobe high even when OCR performance is not perfect.Russell et al [6] note this can, at least in part, beattributed to redundancy and the fact that whileOCR performance may commit some errors, it per-forms very well for English. They go on to discussthe use of handwritten and typed queries. Arabicrecognition technology, however, generally performssignificantly poorer than its English counterpart.

A system for directly searching scanned hand-written English handwriting was discussed in [7].This sytem, known as CEDAR-FOX, was devel-oped for forensic document analysis applications[12],[8]. Searching scanned Arabic text within asystem known as CEDARABIC was first reportedin [9]. The CEDARABIC system was based on anend-to-end software system that was previously de-veloped for English handwriting known as CEDAR-FOX. Both systems are designed to be interactive foruse by a human document examiner and have manypre-processing operations such as line and word seg-mentation, rule-line removal, image enhancement,etc.

3 Queries and Searches

The query can take several forms: (i) a UNICODEstring of Arabic text (for example, entered on anArabic keyboard), specifying a word or words theuser wants to appear in the handwritten document,(ii) an English word or words corresponding to anidea that should appear in the Arabic document,(iii) an image of an Arabic word or words; documentsshould be returned that also have a representationof this Arabic word.

Word spotting algorithms start with an imagequery; either a full word or component prototypecharacters. The document is searched directly, withonly minor preprocessing steps such as noise re-moval, etc. We take two approaches for word spot-ting: word shape and character shape based. Inthe word shape based method, features are extractedfrom prototype word images. This prototype can ei-ther be provided with the query, or can be looked

up from a library of images based on a keyword.We then compare the features of a candidate wordto the prototype words, choosing the best match.The character shape based method splits a candi-date word into sequences of candidate componentcharacters. Each sequence is matched to prototypesof the characters in the query word, and the sequenceof candidate characters containing the overall maxi-mum similarity to the prototype characters receivesthe highest score for that word, with the score act-ing as a confidence measure. The word shape basedmethod performs well when many prototype imagesare present, but cannot be used if there are noneavailable; at that point the character based methodis the only available approach. In situations whereboth methods are applicable, their rankings are com-bined.

To partially transcribe documents we use severalapproaches of word recognition. A baseline, sim-ple method is to perform a variation of characterrecognition and try to directly deduce a word. Ara-bic has an advantage over languages such as Englishbecause of the presence of subwords which are pre-dictably distinct. In a second method, we comparethe candidate characters against those suggested bya lexicon of words, choosing the candidate repre-sentation with the best score. Since larger lexiconsgenerally result in poorer performance we limit thelexicon size when using such a method.

4 Framework

Figure 1 gives an overview of the versatile searchframework. The key point is that both image andtext queries are maintained against image and textversions of the document throughout the searchingprocess, with their results in the end being combinedwith a neural network.

Figure 1: Versatile Search Framework

A critical common preprocessing step necessary

for both methods is segmenting a page into lines,and sometimes a line into words. A CEDARABICrepresentation of a segmented document is shown inFigure 2.

Figure 2: Segmented Document

4.1 Segmentation Algorithms

Automatic word segmentation, as presented in [10],is based on taking several features on either side of apotential segmentation point and using a neural net-work for deciding whether or not the segmentation isbetween two distinct words. Some of the differencesbetween the tasks of segmenting Arabic script andsegmenting Latin script are the presence of multipledots above and below the main body in Arabic andthe absence of upper case letters at the beginningof sentences in Arabic. The method presented wasfound to have an overall correctness of about 60%.

The segmentation free method attempts toperform spotting and segmentation concurrently.Rather than a candidate word image, an entire lineimage acts as input. The line is split into seg-ments based on an algorithm similar to the ligature-based segmentation algorithm used in [4]. All realis-tic combinations of adjacent connected componentsare considered as potential areas where the desiredword may appear. This approach more exhaustivelysearches a line, looking for a given word image, whileat the same time keeping the number of evaluationsmanageable by considering only a small subset ofpotential regions in the image.

4.2 Automatic Word Segmentation

The process of word segmentation begins with ob-taining the set of connected components for eachline in the document image. Figure 3 and Figure 4show the connected components (exterior and inte-rior contours) of a small region of a document im-age. The interior contours or loops in a component

Figure 3: Region of a document image

Figure 4: Connected component exterior and interiorcontours

are ignored for the purpose of word segmentation asthey provide no information for this purpose. Theconnected components are grouped into clusters, bymerging minor components such as dots above andbelow a major component. Also particular to Ara-bic, many words start with the Arabic character“Alef”. The presence of an “Alef” is a strong indi-cator that there may be a word gap between the pairof clusters. The height and width of the componentare two parameters used to check if the componentis the character “Alef”. Figure 7 shows samples ofthe Arabic character “Alef”. Every pair of adjacentclusters are candidates for word gaps. Nine featuresare extracted for these pairs of clusters and a neuralnetwork is used to determine if the gap between thepair is a word gap. The nine features are: width ofthe first cluster, width of second cluster, differencebetween the bounding box of the two clusters, flagset to 1 or 0 depending on the presence or absence ofthe Arabic character “Alef” in the first cluster, thesame flag for the second cluster, number of compo-nents in the first cluster, number of components inthe second cluster, minimum distance between theconvex hulls enclosing the two clusters and the ra-tio between, the sum of the areas enclosed by theconvex hulls of the individual clusters, to the totalarea inside the convex hull enclosing the clusters to-gether. The minimum distance between convex hullsis calculated by sampling points on the convex hullfor each connected component and calculating theminimum distance of all pairs of such points. Figure5 and Figure 6 show two pairs of adjacent clusterstested for word gaps. Dotted lines indicate convexhulls around the individual clusters. Solid lines in-dicate convex hull around the two clusters taken to-gether.

A neural network was trained using these nine fea-

Figure 5: Gap between the convex hulls is not a wordgap

Figure 6: The gap between the convex hulls is a wordgap

tures with the feature vector labeled as to whetherit is a word gap or not. This is similar to the neuralnetwork approach used for English postal addresses[5], but with different features.

4.2.1 Word SegmentationPerformance

When applied to the document set described ear-lier with correctly segmented lines, the overall per-formance is about 60% using a set of seven segmen-tation features. In [10], the authors noted that amore complex set of features is expected to yield ahigher level of performance.

4.3 Segmentation-free LineProcessing

The segmentation free algorithm processes the wordson a per line basis rather than relying on pre-segmented words. The algorithm can be viewed as asequence of steps. First, the image is processed into

Figure 7: Samples of Arabic character “Alef”. Theheight and width are two parameters that are used todetect the presence of “Alef” in the clusters.

component lines. Candidate segmentation points aregenerated for a given line. The line is scanned witha sliding window, generating candidate words andscoring them, as well as filtering out nearly equiva-lent candidates.

4.3.1 Segmentation AlgorithmThe segmentation algorithm used on the line is es-

sentially the same as the one used to generate can-didate character segmentation points in candidatewords in the actual spotting step. It is performedvia a combination of ligatures and concavity featureson an encoded contour of the components of the im-age. Average stroke width is estimated and used todetermine the features.

Ligatures, as noted in [3] are strong candidates forsegmentation points in cursive scripts. Ligatures areextracted in a similar way as in [3]–if the distancebetween y-coordinates of the upper half and lowerhalf of the outer contour for a x-coordinate is lessthan or equal to the average stroke width, then thex-coordinate is marked as an element of a ligature.Concavity features in upper contour and convexi-ties in the lower contour are also used to generatecandidate segmentation points, which are especiallyuseful for distinct characters which are touching, asopposed to being connected. A ligature will causeany overlapped concavity features to be ignored. Fora given x-coordinate, if a concavity and convexityoverlap, a segmentation point is added for that x-coordinate.

While the character based method described in[3] uses this segmentation method to split a wordinto candidate characters, the segmentation free lineprocessing method uses it to split the line, the mo-tivation being to generate candidate word regionson the line. Arabic has predicable breaks in a wordbased on non-connective characters. Therefore, thenumber of connected components in a word is pre-dictable as well.

4.3.2 Line ScanningThe method utilizes a sliding window, starting

from the left of the line and proceeding to the right(in the opposite direction to the way Arabic is writ-ten), although the direction of the scan is unimpor-tant because all realistic combinations of connectedcomponents will be considered.

Each character class c in Arabic is associatedwith a minimum and a maximum durational length(minlen(c) and maxlen(c) respectively). Theselengths are generated by segmenting a representa-tive dataset of characters with the same segmenta-tion algorithm, and taking the min and max for eachcharacter. Due to the nature of the Arabic characterset, the upper bound for all characters is 5, not 4 asin [4].

The scanning algorithm will scan for candidatewords consisting of a range of segments. For agiven search word W of length n, for each characterci ∈ W , the minimum length minlen(W ) consideredis

∑n−1i=0 minlen(ci) and the maximum considered

length maxlen(W ) is∑n−1

i=0 maxlen(ci).The scanning algorithm starts at each segmenta-

tion point p on a line. For a given point pi, if i = 0or if pi.left > pi−1.right (i.e., there is horizontalspace to the left of the segmentation point) it isconsidered a valid start point. Similarly, for a givenpoint pi, if i = max(p) or if pi.right < pi+1.left(i.e., there is horizontal space to the right of thesegmentation point) it is considered a valid end-point. The algorithm considers candidate words tobe ranges of segments between two segmentationpoints pi and pj where pi is a valid start point,minlen(W ) ≤ j − i + 1 ≤ maxlen(W ), and pj isa valid endpoint.

While this generally results in more candidatewords than the other segmentation method, sinceeach Arabic word is only broken into a few piecesseparated by whitespace it does not result in a dra-matic decrease in performance.

4.3.3 FilteringOften, a candidate word influences neighboring

candidate words’ scores. Neighboring candidatewords are those words with overlapping segments.Often, a high scoring word will result in high scoresfor neighbors. The largest issue rises when the highscoring word is, in fact, an incorrect match. In thiscase, the incorrect choice and several of its neigh-boring words receive similarly good scores, pushingthe rank of the actual word lower in the list. An-other issue is if the word being searched for appearsmultiple times in a document. The best matchingwords’ neighboring candidates depresses the secondoccurrence’s rank. Various ways of dealing with theoverlap meet with different degrees of success.

The approach taken in the current incarnation ofthe algorithm is to keep the candidate word thathas the highest score out of the overlapping words.Unfortunately, this occasionally removes the correctword completely from the list. Alternate methods offiltering are being explored.

5 Word Shape Matching

The word segmentation is an indexing step beforeusing the word shape matching for word retrieval.A two-step approach is employed in performing thesearch: (1) prototype selection: the query (Englishtext) is used to obtain a set of handwritten samplesof that word from a known set of writers (these arethe prototypes), and (2) word matching: the pro-totypes are used to spot each occurrence of those

Figure 8: Candidate word regions

words in the indexed document database. A rankingis performed on the entire set of test word images-where the ranking criterion is the mean similarityscore between prototype words and the candidateword based on global word shape features.

5.1 Prototype Selection

Prototypes which are handwritten samples of a wordare obtained from an indexed (segmented) set of doc-uments. These indexed documents contain the truth(English equivalent) for every word image. Such anindexing can be done using a transcript mapping ap-proach such as described in [2]. Synonymous wordsif present in the truth are also used to obtain theprototypes. Hence queries such as “country” will re-sult in selecting prototypes that have been truthedas “country” or “nation” etc... A dynamic program-ming Edit Distance algorithm is used to match thequery text with the indexed word image’s truth.Those with distance as zero are automatically se-lected as prototypes. Others can be selected manu-ally.

5.2 Word Matching

The word matching algorithm uses a set of 1024 bi-nary features for the word images. These binary fea-tures are compared using the correlation similaritymeasure 5.2.1 to obtain a similarity value between 0and 1. This similarity score represents the extent ofmatch between two word images. The smaller thescore, the better is the match. For word spotting,every word image in the test set of documents arecompared with every selected prototype and a distri-bution of similarity values is obtained. The distribu-tion of similarity values is replaced by its arithmeticmean. Now every word is sorted in rank in accor-dance with this final mean score. Figure 9 shows anexample query word image compared with the a setof 4 selected prototypes.

Figure 9: One query word image (left) matched againstfour selected prototype images. The 1024 binary GSCfeatures are shown next to the images.

5.2.1 Similarity measureThe method of measuring the similarity or dis-

tance between two binary vectors is essential. Thecorrelation distance performed best for GSC binaryfeatures [14] which is defined for two binary vectorsX and Y , as in equation 1

d(X, Y ) =1

2

0@1−

s11s00 − s10s01

[(s10 + s11)(s01 + s00)(s11 + s01)(s00 + s10)]12

1A

(1)

where sij represent the number of corresponding bitsof X and Y that have values i and j.

5.3 Word-Matching Performance

The performance of the word spotter was evalu-ated using manually segmented Arabic documents.All experiments and results were averaged over 150queries. For each query, a certain set of documentsare used as training to provide for the template wordimages and rest of the documents were used for test-ing. Figure 14 shows the percentage of times whencorrect word images were retrieved within the topranks scaled on the x-axis. More than one correctword image does exist in the test set. For exam-ple, if the word “king” occurs in document 1 twice,and we use writers 1 through 6 for training, thenwe have 6 ∗ 2 = 12 template word images to per-form word spotting. The remaining test set contains(10− 6) ∗ 2 = 8 correct matches in the test set. Thetop curve of the figure shows that at least one cor-rect word is retrieved 90% of the time within thetop 12 ranks. The other curves show the same whenat least 2 and 3 correct words are retreived. Simi-lar plots can be obtained when different numbers ofwriters are used for training.

Performance of word spotting can be specified interms of precision and recall. If n1 is the number ofrelevant words in the database, n2 is the number ofwords retrieved, and n3 is number of relevant wordsamong those that are retrieved then Precision =n3/n2 and Recall = n3/n1.

6 Text String Matching

The approach is lexicon based; that is, it makes useof the Arabic sequences of characters for words inthe lexicon in order to select sets of prototype im-ages representing the characters forming them. Apreprocessing step is a line and word segmentationprocess, such as the one described in [10]. The can-didate word image is first split into a sequence of seg-ments (as in Figure 10), with the ideal result beingindividual characters in the candidate word beingseparated. The segmentation algorithm used “over-segments” words in the hopes of avoiding incorrectlyputting more than a single character into a segment.Segments are then rejoined and features extracted,which are in turn compared to features of prototypeimages of the characters. Further issues of underseg-menting unique to Arabic are dealt with using com-pound character classes. A score that represents thematch between the lexicon and the candidate wordimage is then computed. The score relates to the in-dividual character recognition scores for each of thecombined segments of the word image.

Figure 10: Arabic word “king”, with segmentationpoints and corresponding best segments shown below

6.1 Character Dataset

Unlike the method presented in [10], this method de-pends on character images rather than word imagesto form a basis for comparison. In this method, theimage is essentially cut into component charactersand each character is matched for similarity, in con-trast to the latter’s method of matching entire wordshape. Since the number of characters in Arabic is arather small compared with the number of potentialwords, a library of component characters can be in-corporated directly into the system. This eliminatesthe indexing phase of [10].

As such a character database was not readily avail-able, a new character image dataset was derivedfrom the existing Arabic document dataset producedfrom the CEDARABIC [9] project. The originaldataset consisted of a collection of handwritten doc-

uments produced from a variety of authors and isdescribed in Section 7.1. The scanned words wereindividualized and raw ASCII descriptions of theArabic characters were given corresponding to thecontent of the word. The derived dataset consistsof images of single Arabic characters and charactercombinations. Approximately 2,000 images of char-acters and character combinations in other configu-rations were created by allowing the ligature basedsegmentation algorithm to create candidate super-segments of the truthed words, and manually match-ing the best candidate supersegments to the corre-sponding character or character combination whenthe segmentation was successful. Both left to rightand right to left versions of the writings were tested,the original right to left images producing better re-sults (left to right versions occasionally seemed tobe more prone to undersegmenting the words). The2,000 images represent but a small fraction of po-tential images from this dataset. Work on extendingthis dataset is ongoing.

6.2 Features

WMR features for each of the character imageswere extracted and incorporated into the recogni-tion engine of CEDARABIC. As described in [11],the WMR feature set consist of 74 features. Twoare global features–aspect and stroke ratio of theentire character. The remaining 72 are local fea-tures. Each character image is divided into 9 subim-ages. The distribution of the 8 directional slopesfor each subimage form this set (8 directional slopes× 9 subimages = 72 features). Fli,j = si,j/NiSj ,i = 1, 2, ..., 9, j = 0, 1, ..., 7, where si,j = numberof components with slope j from subimage i, whereNi = number of components from subimage i, andSj = max(si,j/Ni). These features are the basisof comparison for the character images derived fromthe segmentation of words to be recognized. To date,this appears to be the first application of WMR fea-tures to Arabic recognition.

To obtain preliminary results, the base shape of aletter was mapped to all derivations of that letter.For example, the base shape of the character beh wasmapped to beh, teh, and theh. The initial and me-dial forms of beh were also mapped to the initial andmedial forms of noon and yeh. If separately truthedversions were available specifying explicit member-ship to, for example, teh, such characters were onlyincluded in teh’s set of features.

6.3 Image Processing AndSegmentation

Image processing is accomplished via a method sim-ilar in part to that described in [3]. First, a chaincode representation of the binary image’s contours

is generated. Noise removal, slant correction, andsmoothing is performed. Segmentation is performedvia a combination of ligatures and concavity featureson an encoded contour of the components of the im-age. Average stroke width is estimated and used todetermine the features. The number of segmenta-tion points is kept to a minimum, but unlike in [3],the maximum number of segmentation points percharacter is five. WMR features are extracted fromsegments.

One goal of this method is to oversegment wordswith the hopes of eliminating undersegmentation al-together. Undersegmentation in ligature based seg-mentation of Arabic text, however, continues to beproblematic due to the presence of character com-binations and vertically separated characters. Forexample, some writing styles do not mark certainletters with much clarity–especially initial charac-ters, for example initial yeh’s. Since the ligaturebased segmentation proceeds horizontally, seekingbreaking points at various positions along the x-axis,the vertical “stacking” of characters cannot be dealtwith simply by increasing the sensitivity of the seg-mentation (see Figure 11). To deal with these issues,character classes were defined corresponding to thecommon character and vertically occurring combi-nations.

Figure 11: Left: (horizontally) unsegmentable charactercombination, Right: individual images

6.3.1 Preprocessing Lexicon

An Arabic word is specified as a sequence ofthe approximately 28 base letters. To aid recog-nition, a simple algorithm maps the given text tothe correct variation of each character. For exam-ple, “Alef|Lam|Teh|Qaf|Alef maksura|” is mapped to“Alefi|Lami|Tehm|Qafm|Alef maksuraf |” where “i”means the letter is in the initial position, “m” meansthe letter is in the medial position, etc. Additionalpost-processing steps to the Arabic lexicon combineadjacent individual characters in appropriate posi-tions into character combination classes. For exam-ple, “Lami|Meemm|” is mapped to “Lammeemi|.”

The new mapping system for the 150 new char-acter classes were incorporated into the charac-ter recognition model of CEDARABIC, replacingthe support for English letters carried over fromCEDAR-FOX.

6.3.2 Word RecognitionThe objective is to find the best match between

the lexicon and the image. In contrast to [3], up tofive adjacent segments are compared to the charac-ter classes dictated as possibilities by a given lexi-con entry. In the first phase of the match, the mini-mum Euclidean distance between the WMR featuresof candidate supersegments and the prototype char-acter images is computed. In the second phase, aglobal optimum path is obtained using dynamic pro-gramming based on the saved minimum distancesobtained in the first matching phase. The lexicon isranked, the entries with the lowest total scores beingthe closest matches. Figure 10 shows the the Arabicword “king” split into its best possible segments.

Testing proceeded on the same 10 authors’ doc-uments as in [10]. Recognition was attempted onapproximately 180 words written by each of the 10authors (for a total of approximately 1,800 words).Recognition was attempted in two runs, one with alexicon size of 20 words and one with a size of 100.The lexicon was generated from other words amongthe 180 being recognized. The words and the lexi-cons in the tests were the same for all authors.

6.4 Word Spotting

Word spotting proceeds in a very similar fashion toword recognition. In the case of word spotting, thelexicon consists only of the word being spotted. Ascore against this lexicon entry is generated for eachcandidate word in the document. The candidatewords are ranked according to score, the words withthe best scores are most likely to be the word beingspotted.

From the documents written by the authors, 32words were chosen at random and “spotted,” in asimilar fashion to the experiments performed in [10].Note that the recall for word spotting when utilizingthe expanded Arabic character classes is nearly 80%for a precision of 50, which is a significant improve-ment over the other methods (using simply the Ara-bic letters individually and the image based methoddescribed in [10]).

7 Results

7.1 Document Image Database

For evaluating the results of our methods, we used adocument collection prepared from 10 different writ-ers, each contributing 10 different full page docu-ments in handwritten Arabic. Each document iscomprised of approximately 150 − 200 words each,with a total of 20, 000 word images in the entire data-base. The documents were scanned at a resolutionof 300 dots per inch. This is the resolution to beused for optimal performance of the system. Fig-

Figure 12: Sample “.arb” file opened with CEDARA-BIC. Each segmented word is colored differently fromadjacent words.

ure 12 shows a sample scanned handwritten Arabicdocument written by writer 1.

For each of the 10 documents that were handwrit-ten, a complete set of truth values, comprised of thealphabet sequence, meaning, and the pronunciationin that document was also given. The scanned hand-written documents’ word images were mapped withthe corresponding truth information (see Figure 13).

Figure 13: Word image with corresponding truth

7.2 Experiments

To test the combined method, 32 queries were issuedon 13,631 records. The neural network was trainedon 300 such queries, 150 positive and 150 negativequery matches. All remaining records were used fortesting. The combined score is a score between -1and 1. A negative score would indicate a mismatchand a positive score a match. A 91% raw classifica-tion accuracy was observed.

Using five writers for providing prototypes andthe other five for testing, using manually segmenteddocuments, 55% precision is obtained at 50% recall

for the word shape method alone. The characterbased method achieves 75% precision at the samerecall rate. The combined method is consistentlybetter, resulting in about 80% precision. A compar-ison graph, with the word shape method using fivewriters, is shown in Figure 14. One search resultfrom CEDARABIC is shown in Figure 15.

10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100precision recall

Recall

Pre

cisi

on

Word ShapeWMRCombination

Figure 14: Precision-recall comparison of word-shape, character-shape and combination approaches.

8 Conclusion and Future Directions

Processing image and text based queries in parallelcan result in higher performance than either alone.The versatile search framework presented can be ap-plied to many document search problems. The ex-ample presented illustrates a word spotting appli-cation, but other document search strategies mayexperience similar performance increases. For exam-ple, a partially transcribed document could be repre-sented as a “bag of words,” with a term/documentmatrix. From there, latent semantic analysis TF-IDF (term frequency-inverse document frequency)

Figure 15: Word Spotting Testing Results

weights can be used to perform traditional searches,with image search augmenting less than perfect tran-scription techniques. Furthermore, “plugging in”improved image or text-based search algorithms canpush overall performance higher. For our experi-ments, we used the neural network to simply weightthe two incoming scores. However, features ex-tracted from the images may improve neural networkperformance.

References

[1] D. Hand, H. Mannila, and P. Smyth. Principlesof Data Mining. MIT Press, Cambridge, MA,2001.

[2] C. Huang and S. N. Srihari. Mappingtranscripts to handwritten text. In Proc.Tenth International Workshop on Frontiers inHandwriting Recognition (IWFHR)), La Boule,France, 2006. IEEE Computer Society.

[3] G. Kim. Recognition of offline handwrittenwords and extension to phrase recognition. Doc-toral Dissertation, State University of NewYork at Buffalo, 1997.

[4] G. Kim and V. Govindaraju. A lexicon drivenapproach to handwritten word recognition forreal-time applications. In IEEE Transactionson Pattern Analysis and Machine Intelligence19(4), pages 366–379, 1997.

[5] G. Kim, V. Govindaraju, and S. N. Srihari.A segmentation and recognition strategy forhandwritten phrases. In International Confer-ence on Pattern Recognition, ICPR-13, pages510–514, 1996.

[6] G. Russell, M.P. Perrone, Yi min Chee, and Air-nan Ziq. Handwritten Document Retrieval. InProc. Eighth International Workshop on Fron-tiers in Handwriting Recognition, pages 233–238, Niagara-on-the-lake, Ontario, 2002.

[7] S. N. Srihari, C. Huang, and H. Srinivasan. Asearch engine for handwritten documents. InDocument Recognition and Retrieval XII: Pro-ceedings SPIE, pages 66–75, San Jose, CA,2005.

[8] S. N. Srihari and Z. Shi. Forensic handwrittendocument retrieval system. In Proc. DocumentImage Analysis for Libraries (DIAL), pages188–194, Palo Alto, CA, 2004. IEEE ComputerSociety.

[9] S. N. Srihari, H. Srinivasan, P. Babu, andC. Bhole. Handwritten Arabic word spotting

using the CEDARABIC document analysis sys-tem. In Proc. Symposium on Document ImageUnderstanding Technology (SDIUT-05), pages123–132, College Park, MD, Nov. 2005.

[10] S. N. Srihari, H. Srinivasan, P. Babu, andC. Bhole. Spotting words in handwritten Arabicdocuments. In Document Recognition and Re-trieval XIII: Proceedings SPIE, pages 606702–1to 606702–12, San Jose, CA, 2006.

[11] S. N. Srihari, C. I. Tomai, B. Zhang, and S. Lee.Individuality of numerals. In Proc. SeventhInternational Conference on Document Analy-sis and Recognition(ICDAR), page 1096, Edin-burgh, UK, 2003. IEEE Computer Society.

[12] S. N. Srihari, B. Zhang, C. Tomai, S. Lee,Z. Shi, and Y-C. Shin. A search engine forhandwritten documents. In Proc. Symposiumon Document Image Understanding Technol-ogy (SDIUT-05), pages 67–75, Greenbelt, MD,2003.

[13] K. Taghva, J. Borsack, and A. Condit. Resultsof applying probabilistic IR to OCR text. InResearch and Development in Information Re-trieval, pages 202–211, 1994.

[14] B. Zhang and S. N. Srihari. Binary vector dis-simularity measures for handwriting identifica-tion. Proceedings of the SPIE, Document Recog-nition and Retrieval, pages 155–166, 2003.