Occluded Character Restoration Using Active Contour withShape Priors

Rafi Cohen, Klara Kedem, Itshak Dinstein, Jihad El-SanaBen-Gurion University, Beer-Sheva, Israel

{rafico,el-sana,klara}@cs.bgu.ac.il, dinstein@ee.bgu.ac.il

Abstract

Broken or partially visible characters is com-mon phenomenon in historical documents. It stemsfrom various factors, such as overlaid text or degra-dation. Restoring such characters is necessaryfor document analysis applications. This paperpresents a new approach for restoring underlayingHebrew broken characters that were partially oc-cluded by Arabic text in a palimpsest. We applytext recognition on the fragments of the Hebrew let-ters and select the k candidate letters that matchbest the fragments of the Hebrew letters. We thencomplete the broken Hebrew characters using activecontour with the k candidates as shape priors. Weuse a modified geodesic active contour, which wetailored to occluded text restoration. It is initializedon the fragments of the Hebrew text, then it under-goes an expansion phase and a contraction phasevia the occluding Arabic text to form the restoredHebrew character. We measure the distance be-tween the completed character and its correspondingpriors and choose the shape with the minimal dis-tance as the reconstructed character. Experimentalresults are presented. On average 68% of the char-acters were correctly reconstructed.

1 Introduction

Parchment was used in the middle ages as oneof the main writing materials. Due to its high cost,the writing was often scraped off or washed to bereused. Documents consisting of reused parchmentsare called palimpsests. Many palimpsests are valu-able, as they may contain, under the overlaid text,an important historical information. A well knownexample is the Archimedes palimpsest [14, 20]. Theoriginal text in palimpsests is often broken, due tothe occlusion caused by the more recent writing.

In this paper we address the problem of restor-ing broken characters caused by occlusion. Morespecifically, we address the restoration of Hebrewtext that was partially occluded by Arabic text(Figure 1).

The research in restoration of occluded and bro-ken characters could be classified to three main ap-proaches: in-painting, active contours, and rule-based systems. Next, we briefly overview relatedwork in each category.

In-painting is defined as the automatic fillingof an unknown image region [5]. Most of in-painting algorithms are for natural images and donot fit well text images. Bertozzi et al. [6] sug-gest an in-painting method for text images that isbased on a slightly modified Cahn-Hilliard equa-tion, which models the phase separation in binaryfluids. Chang et al. [10] address the problem of vis-ible watermark characters, occluded by foregroundcontent. Their algorithm is based on an exemplar-based in-painting which reconstructs the missingareas by referring to similar patches from undam-aged areas. Hollaus et al. [13] perform in-paintingof the Archimedes palimpsest using a high-orderMarkov random field. These in-painting methodsdo not use character models for the reconstructionand this causes smearing in the in-painted area, andinability to restore severely broken characters. Al-lier et al. [1] restore broken printed Latin charac-ters using the classical active contour model, with aGVF field and a character model. The model is au-tomatically selected using a bank of Gabor filters,and the completion is done in the degraded area.Droettboom [11] propose a method based on graphcombinatorics to merge broken characters of histor-ical documents. The goal of his algorithm is to findan optimal way to join connected components ona given page that would maximize the mean con-fidence of all characters. Several approaches forrestoring broken digits in binary images extend the

2012 International Conference on Frontiers in Handwriting Recognition

978-0-7695-4774-9/12 $26.00 © 2012 IEEE

DOI 10.1109/ICFHR.2012.243

495

2012 International Conference on Frontiers in Handwriting Recognition

978-0-7695-4774-9/12 $26.00 © 2012 IEEE

DOI 10.1109/ICFHR.2012.243

497

boundaries of the character using masks, morpho-logical operators and region-growing [19, 22, 21].

We use an active contour with shape prior torestore the underlaid text in palimpsests. In thiswork, we assume that the input consists of seg-mented Arabic text (the overlaid layer) and par-tially occluded Hebrew text (the underlaid layer).We apply our text recognition algorithm [17] tofind a constant number, k, of candidate letters thatmatch best the fragments of the Hebrew letters. Wethen complete the broken Hebrew characters usinga 2-phase active contour for occluded text (more inSection 3) with the k candidates as shape priors.

We developed a 2-phase active contour for oc-cluded text. We compared the results of our ac-tive contour with the k candidate letters using theshape context algorithm [4] and chose the shapewith the minimal distance as the reconstructed He-brew character.

The paper is organized as follows, In Section 2we describe briefly active contours with shape prior,then in Section 3 we describe our reconstructionapproach followed by experimental results in Sec-tion 4. Finally, we suggest directions for futurework in Section 5.

2 Active contours with shape prior

Active contours have been used successfully fordetection and segmentation in various image pro-cessing applications [7, 18, 16, 9]. The geodesicactive contour (GAC) model [8] is an edge basedactive contour, in which a curve is evolving to min-imize the following energy term:

EGAC =

∫g(|∇I(C(s))|)ds, (1)

where g is an inverse edge indicator function (e.g.,g(I) = 1/(1 + |∇I|2)), and C is the evolving curve.The steady state of the active contour is reached byevolving each contour point according to Eq. (2),

where κ is the contour curvature, �N is the unitnormal to the curve, and ν is a real constant, whichcontrols the inflation and shrinking of the contour;e.g. negative value of ν causes inflation.

∂C

∂t= g(I)(ν + κ) �N − (∇g · �N) �N, (2)

The implementation of the geodesic active con-tour is based on the level set framework [15]. Themain idea of the level set method is to embed the

evolving contour C as the zero level set of an im-plicit function φ, defined in a higher dimensionC(t) = {(x, y)|φ(x, y, t) = 0}.

In order to integrate a shape as a prior knowl-edge to the segmentation process, it is necessary torepresent the shape as a signed distance function(SDF) [16, 18]. In the level-set framework, inte-grating the shape becomes natural. Let φ be theSDF for the segmentation, and ψ be the SDF ofthe embedding shape. Then, their shape differencecan be defined as in equation (3), where H(x) isthe Heaviside step function [9].

Eshape =

∫(H(φ)−H(ψ))2dx, (3)

3 Our approach

Our restoration method works at character leveland aims to restore Hebrew characters occluded byArabic text in palimpsests. The algorithm consistsof four steps which are discussed next.

3.1 Character fragment extraction

In the first step of the algorithm we compute theboundaries of the connected components which arefragments of the Hebrew characters. To segmentthese fragments, we initialize an inflating geodesicactive contour on the fragments, and set the pixelsof the foreground Arabic letters as barriers. We al-low the active contour to evolve until it convergesto a stable solution at the boundary of the frag-ments of characters (Figure 3(c)). Once the char-acter fragments are detected, we manually assign abounding box for each character to group the frag-ments of the occluded character.

3.2 Preliminary recognition

For each document we create a training set ofmodels consisting of some completely visible repre-sentatives of characters in the text document (notall the characters are occluded). The recognition ofa partially occluded character, c, is performed bycomputing the distance between each of the mod-els and c. The closest k models are selected aspotential candidates for c. The score of matchingthe partially occluded character, c, and a candi-date model, m, is computed by the GSC methodin [12, 17],which employs normalized features andflexible window sizes.

496498

(a) (b)

Figure 1. (a) A Patch from the Palimpsest “L 120 SUP c. 7”, and (b) its corresponding seg-mented Hebrew And Arabic texts in white and green texts, respectively.

3.3 Model set selection for a character

To reduce the computational cost of the recon-struction using shape-prior, we use a method simi-lar to the one suggested by Bar-Yosef et al. [3], witha different way for picking the representative mod-els. For each character c we manually build a setof reference models that represent the variationsamong the different shapes of c in the document.We rearrange the signed distance function (SDF)of the training characters as column vectors, andapply principal component analysis (PCA) to re-duce dimensionality [3]. In this paper, we use thefirst N principal components and cluster the N -dimensional vectors into k clusters using k-means.For each cluster, we choose the model closest tothe mean as the representative shape model (of thecluster). Figure 2(a) shows a set of 34 charactersfrom which we choose five models (see Figure 2(b)).

3.4 Opening barriers for a character

The development of the modified geodesic ac-tive contour (GAC) for character reconstruction isguided by a shape-prior energy (see Eq. (4)). Theenergy of the prior’s shape, EShape, stays constantduring the development of the active contour, whilethe EGAC is changing according to the phase of thealgorithm.

E = EGAC + EShape (4)

To complete the reconstruction of the Hebrew

(a) (b)

Figure 2. (a) A training set of 34 alephcharacters; (b) the 5 representatives cho-sen by PCA and clustering.

497499

(a) (b) (c)

(d) (e) (f)

Figure 3. Illustration of the steps in our al-gorithm. (a) input; (b) the chosen shape-prior; (c) the edges of the seen Hebrewletters, as computed by the geodesic ac-tive contour, with the Arabic letters asbarriers; (d) the active contour breachingthe barriers into the arabic component.(e) the maximal inflation of the active con-tour; and (f) the computed borders of theHebrew text are shown.

letters we remove the barriers caused by the Ara-bic components (the foreground content), by dis-carding the external forces on the contour insidethe Arabic characters. This enables the Hebrewfragments to expand and merge with each otherthrough the occluding Arabic text. This is done bysetting the input of the inverse indicator function,g, from Eq. (2) to g(IA + IH), where IA is definedas:

IA(x, y) =

{255, (x, y) is an Arabic pixel,0, otherwise ,

(5)

and IH is defined in a similar way. We also set thevalue of ν (Eq. (2)), which controls the inflationarybehaviour of the active contour, to a negative value.

Once the number of connected components gen-erated by the active contour equals the number ofconnected components of the character model, thealgorithm switches phase. In the second phase itdiscards the inflating force and the contour startsto shrink to its final shape (the prior). This is doneby setting the value of ν to zero in pixels that belongto the Arabic components. Figures 3(a) and 3(b)show the input dataset and the selected prior, re-spectively. Figures 3(c)-(f) illustrate the develop-ment of the active contours.

3.5 Final recognition

In our implementation we use k models and Nclusters. Hence, the reconstruction using shape-prior is performed kN times for each broken letter.The distance between the reconstructed shape, andthe corresponding prior is measured using shapecontext [4]. We choose the shape with the minimaldistance as the reconstructed character.

4 Experimental results

We experimented with our approach using vari-ous palimpsests from the Arabic manuscripts of theAmbrosiana from Milan library. We generated theground truth (GT) data by using Bar-Yosef’s bi-narization method [2] to segment the Arabic char-acters. The segmentation of the Hebrew characterfragments was done manually and was verified byexperts in Jewish history.

A total number of 765 occluded characters wererestored using the proposed algorithm. In our im-plementation we set N = k = 5, and measured therestoration accuracy using the Normalized Sym-metric Difference (NSD) metric (see Eq. 6, where L

is the ground-truth character shape, and L̂ the re-stored character). We used |S| to denotes the num-ber of pixels contained in the shape S. Note thatNDS is a dissimilarity measure given in percentage,i.e., lower values correspond to higher restorationaccuracy.

NSD =|L\L̂|+ |L̂\L|

|L| × 100 (6)

Figure 6 reports the average recognition ratesand NSD for Hebrew characters from differentdatasets. The overall percentage of occluded char-acters which we correctly recognized is 67.58%, andthe average NSD is 8.28%. This low NSD value in-dicates that the restored character has high similar-ity with the ground-truth character. In computingthe statistics, we omitted the Hebrew letters ן,ו,י

which are almost indistinguishable in the Hebrewhandwriting. In the future we plan to use a dictio-nary to improve the recognition rate of these letters.An examination of the unrecognized characters, re-veals that 68.26% of the unrecognized charactersfailed to be recognized in the preliminary recogni-tion [17]. That is, none of the 5 candidate charac-ters suggested was the ground truth character. Foran analysis of the possible reasons for that we referthe reader to [17]. Figure 5 illustrates the results

498500

(a) (b) (c) (d) (e)

Figure 4. Normalized Symmetric Differ-ence: (a) the occluded character; (b) theground-truth; (c) the prior; (d) the restora-tion result; and (e) finally the ground-truth(Red) and the restored character (Yellow)overlaid together. The NSD values for theupper and lower rows are 15% and 8%, re-spectively.

(a) Input

(b) Hebrew ground truth

(c) Result of proposed method

Figure 5. Results for “L 120 SUP c. 7”palimpsest.

of our algorithm on a text line, and Figure 4 de-picts the computation of the NSD on two differentHebrew characters.

5 Conclusions and future work

We have presented an approach restoring bro-ken Hebrew characters, using active contours withshape-prior. We presented a simple scheme forchoosing the correct shape prior.

Our future work will focus on improving someaspects of the algorithm. In our current implemen-tation, the bounding box of character fragments(that belong to the same letter) is manually cho-sen. We intend to automate this process by firstdividing the document into lines, and then to ap-ply a sliding window over each line. The size of the

window is determined by the range of character di-mensions in the document. A classifier will be usedto determine whether the fragments in the windowsbelong to one character or not. We also intend toexperiment with different classifiers.

Acknowledgment

This research was supported in part by the Is-rael Science Foundation grant no. 1266/09, DFG-Trilateral Grant no. 8716, the Lynn and WilliamFrankel Center for Computer Sciences, and thePaul Ivanier Center for Robotics and ProductionManagement at Ben-Gurion University, Israel. Wewould like to thank Prof. Uri Ehrlich and Uri Safraifrom the Goldstein-Goren department of Jewishthought, Ben-Gurion University of the Negev, fortheir assistance in generating the ground truth.The authors also thank Irina Rabaev for providingus the source code of her algorithm.

References

[1] B. Allier, N. Bali, and H. Emptoz. Auto-matic accurate broken character restorationfor patrimonial documents. InternationalJournal on Document Analysis and Recogni-tion, 8(4):246–261, 2006.

[2] I. Bar-Yosef, I. Beckman, K. Kedem, andI. Dinstein. Binarization, character extrac-tion, and writer identification of historical he-brew calligraphy documents. InternationalJournal on Document Analysis and Recogni-tion, 9(2):89–99, 2007.

[3] I. Bar-Yosef, A. Mokeichev, K. Kedem,I. Dinstein, and U. Ehrlich. Adaptive shapeprior for recognition and variational segmen-tation of degraded historical characters. Pat-tern Recognition, 42(12):3348–3354, 2009.

[4] S. Belongie, J. Malik, and J. Puzicha. Shapematching and object recognition using shapecontexts. Pattern Analysis and Machine In-telligence, IEEE Transactions on, 24(4):509–522, 2002.

[5] M. Bertalmio, G. Sapiro, V. Caselles, andC. Ballester. Image inpainting. In Proceedingsof the 27th annual conference on Computergraphics and interactive techniques, pages417–424. ACM Press/Addison-Wesley Pub-lishing Co., 2000.

[6] A. Bertozzi, S. Esedoglu, and A. Gillette. In-painting of binary images using the cahn–hilliard equation. Image Processing, IEEETransactions on, 16(1):285–291, 2007.

499501

(a) Occurrences (b) NSD

Figure 6. Statistical results for the palimpsests. (a) dark gray represents the number of occur-rences of the Hebrew character on the x-axis and light gray the number of successful recogni-tions; (b) the NSD for the Hebrew letters on the x-axis.

[7] A. Blake and M. Isard. Active Con-tours: The Application of Techniques fromGraphics,Vision,Control Theory and Statis-tics to Visual Tracking of Shapes in Motion.Springer, 1998.

[8] V. Caselles, R. Kimmel, and G. Sapiro.Geodesic active contours. International jour-nal of computer vision, 22(1):61–79, 1997.

[9] T. Chan and W. Zhu. Level set based shapeprior segmentation. In Computer Visionand Pattern Recognition, 2005. CVPR 2005.IEEE Computer Society Conference on, vol-ume 2, pages 1164–1170. IEEE, 2005.

[10] L. Chang, J. Sun, M. Suwa, H. Takebe, Y. He,and S. Naoi. Occluded text restoration andrecognition. In Proceedings of the 9th IAPRInternational Workshop on Document Anal-ysis Systems, pages 151–158. ACM, 2010.

[11] M. Droettboom. Correcting broken charac-ters in the recognition of historical printeddocuments. In Proceedings of the 3rdACM/IEEE-CS joint conference on Digitallibraries, pages 364–366. IEEE Computer So-ciety, 2003.

[12] J. Favata and G. Srikantan. A mutiplefeature/resolution approach to handprintedcharacter/digit recognition. InternationalJournal of imaging Systems and technology7, pages 304–311, 1996.

[13] F. Hollaus and R. Sablatnig. Inpaint-ing of occluded regions in handwritings.OAGM/AAPR Workshop, Austrian Com-puter Society, 2011.

[14] R. Netz and W. Noel. The Archimedes Codex.Weidenfeld & Nicolson, 2007.

[15] S. Osher and J. Sethian. Fronts propagatingwith curvature-dependent speed: algorithmsbased on hamilton-jacobi formulations. Jour-

nal of computational physics, 79(1):12–49,1988.

[16] N. Paragios, M. Rousson, and V. Ramesh.Matching distance functions: a shape-to-areavariational approach for global-to-local regis-tration. European Conference on ComputerVision (ECCV), pages 813–815, 2002.

[17] I. Rabaev, O. Biller, J. El-Sana, K. Kedem,and I. Dinstein. Features for hebrew charac-ter searching. Technical Report 12-07, Ben-Gurion University of the Negev, May 2012.

[18] M. Rousson and N. Paragios. Shape priorsfor level set representations. European Con-ference on Computer Vision (ECCV), pages416–418, 2002.

[19] Z. Shi and V. Govindaraju. Character im-age enhancement by selective region-growing.Pattern recognition letters, 17(5):523–527,1996.

[20] The Archimedes Palimpsest Project. http:

//archimedespalimpsest.org/.[21] A. Whichello and H. Yan. Linking broken

character borders with variable sized masksto improve recognition. Pattern Recognition,29(8):1429–1435, 1996.

[22] D. Yu and H. Yan. Reconstruction of bro-ken handwritten digits based on structuralmorphological features. Pattern Recognition,34(2):235–254, 2001.

500502