Image Inpainting

MarceloBertalmioandGuillermoSapiro�

ElectricalandComputerEngineering,Universityof MinnesotaVicentCasellesandColomaBallester

EscolaSuperiorPolitecnica,UniversitatPompeuFabra

Abstract

Inpainting,thetechniqueof modifyinganimagein anundetectableform, is asancientasart itself. The goalsandapplicationsof in-paintingarenumerous,from the restorationof damagedpaintingsandphotographsto theremoval/replacementof selectedobjects.Inthis paper, we introducea novel algorithmfor digital inpaintingofstill imagesthatattemptsto replicatethebasictechniquesusedbyprofessionalrestorators.After the userselectsthe regions to berestored,thealgorithmautomaticallyfills-in theseregionswith in-formationsurroundingthem.Thefill-in is donein sucha way thatisophotelinesarriving at theregionsboundariesarecompletedin-side. In contrastwith previous approaches,the techniqueherein-troduceddoesnot requirethe userto specifywherethe novel in-formationcomesfrom. This is automaticallydone(and in a fastway), therebyallowing to simultaneouslyfill-in numerousregionscontainingcompletelydifferent structuresand surroundingback-grounds.In addition,no limitationsareimposedon thetopologyofthe region to be inpainted. Applicationsof this techniqueincludetherestorationof oldphotographsanddamagedfilm; removal of su-perimposedtext like dates,subtitles,or publicity; andthe removalof entireobjectsfrom the imagelike microphonesor wires in spe-cial effects.

CR Categories: I.3.3 [Computer Graphics]: Picture/ImageGeneration—;I.3.4 [ComputerGraphics]: GraphicsUtilities—; I.4.4 [Image Processingand ComputerVision]: Restoration—;I.4.9 [ImageProcessingandComputerVision]: Applications—;

Keywords: Imagerestoration,inpainting,isophotes,anisotropicdiffusion.

1 Introduction

The modificationof imagesin a way that is non-detectablefor anobserver whodoesnot know theoriginal imageis a practiceasoldasartisticcreationitself. Medieval artwork startedto berestoredasearlyastheRenaissance,themotivesbeingoftenasmuchto bringmedieval pictures“up to date” as to fill in any gaps[1, 2]. Thispracticeis calledretouchingor inpainting. Theobjectof inpaintingis to reconstitutethe missingor damagedportionsof the work, inorderto make it morelegibleandto restoreits unity [2].

The needto retouchthe imagein an unobtrusive way extendednaturally from paintingsto photographyand film. The purposesremainthesame:to revertdeterioration(e.g.,cracksin photographsor scratchesanddustspotsin film), or to addor remove elements(e.g.,removal of stampeddateandred-eye from photographs,theinfamous“airbrushing”of political enemies[3]).

Digital techniquesarestartingto beawidespreadwayof inpaint-ing, rangingfrom attemptsto fully automaticdetectionandremovalof scratchesin film [4, 5, 6], all thewayto softwaretoolsthatallowasophisticatedbut mostlymanualprocess[7].

In thisarticlewe introduceanovel algorithmfor automaticdigi-tal inpainting,beingits mainmotivationto replicatethebasictech-�

ElectricalandComputerEngineering,University of Minnesota,Min-neapolis,MN 55455,USA,

�marcelo,guille� @ece.umn.edu

niquesusedby professionalrestorators.At thispoint, theonly userinteractionrequiredby the algorithm hereintroducedis to markthe regionsto be inpainted.Althougha numberof techniquesex-ist for the semi-automaticdetectionof imagedefects(mainly infilms), addressingthis is out of thescopeof this paper. Moreover,sincethe inpaintingalgorithmherepresentedcanbe usednot justto restoredamagedphotographsbut alsoto remove undesiredob-jectsandwritingsontheimage,theregionsto beinpaintedmustbemarkedby theuser, sincethey dependon his/hersubjective selec-tion. Herewe areconcernedon how to “fill-in” the regionsto beinpainted,oncethey have beenselected.1 Marked regionsareau-tomaticallyfilled with thestructureof their surrounding,in a formthatwill beexplainedlaterin thispaper.

2 Related work and our contribution

We shouldfirst notethat classicalimagedenoisingalgorithmsdonot applyto imageinpainting,sincetheregionsto beinpaintedareusuallylarge. Thatis, regionsoccupiedby top to bottomscratchesalong several film frames,long cracksin photographs,superim-posedlargefonts,andsoon, areof significantlargersizethanthetypeof noiseassumedin commonimageenhancementalgorithms.In addition,in commonimageenhancementapplications,thepixelscontainbothinformationabouttherealdataandthenoise(e.g.,im-ageplusnoisefor additive noise),while in imageinpainting,thereis no significant information in the region to be inpainted. Theinformation is mainly in the regions surroundingthe areasto beinpainted. Thereis thena needto develop specifictechniquestoaddresstheseproblems.

Mainly threegroupsof workscanbe found in the literaturere-latedto digital inpainting. Thefirst onedealswith the restorationof films, thesecondoneis relatedto texturesynthesis,andthethirdone,a significantlylessstudiedclassthoughvery influential to thework herepresented,is relatedto disocclusion.

Joyeuxet al. [4] deviseda 2-stepsfrequency basedreconstruc-tionsystemfor detectingandremoving linescratchesin films. Theyproposeto first recover low andthenhigh frequencies.Althoughgood resultsare obtainedfor their specificapplication,the algo-rithmcannothandlelargelossareas.Frequency domainalgorithmstradeagoodrecoveryof theoverallstructureof theregionsfor poorspatialresultsregarding,for instance,thecontinuityof lines.

Kokaramet al. [6] usemotion estimationand autoregressivemodelsto interpolatelossesin films from adjacentframes. Thebasicideais to copy into thegaptheright pixels from neighboringframes.Thetechniquecannotbeappliedto still imagesor to filmswheretheregionsto beinpaintedspanmany frames.

HiraniandTotsuka[8] combinefrequency andspatialdomainin-formationin orderto fill agivenregionwith aselectedtexture.Thisis averysimpletechniquethatproducesincrediblegoodresults.Onthe otherhand,the algorithmmainly dealswith texture synthesis

1In orderto studytherobustnessof thealgorithmhereproposed,andnotto betoodependenton themarkingof theregionsto beinpainted,we markthemin avery roughform with any availablepaintbrushsoftware.Markingtheseregionsin theexamplesreportedin thispaperjust takesafew secondsto anon-expertuser.

(andnotwith structuredbackground),andrequirestheuserto selectthete� xtureto becopiedinto theregion to beinpainted.For imageswheretheregion to bereplacedcoversseveraldifferentstructures,theuserwouldneedto gothroughthetremendousworkof segment-ing themandsearchingcorrespondingreplacementsthroughoutthepicture. Although part of this searchcan be doneautomatically,this is extremelytimeconsumingandrequiresthenon-trivial selec-tion of many critical parameters,e.g.,[9]. Othertexturesynthesisalgorithms,e.g.,[9, 10, 11], canbeusedaswell to re-createa pre-selectedtextureto fill-in a (square)region to beinpainted.

In the group of disocclusionalgorithms,a pioneeringwork isdescribedin [12]. Theauthorspresenteda techniquefor removingocclusionswith the goal of imagesegmentation.2 The basicideais to connectT-junctionsat thesamegray-level with elasticamin-imizing curves. The techniquewas mainly developedfor simpleimages,with only a few objectswith constantgray-levels,andwillnot be applicablefor the exampleswith naturalimagespresentedlaterin this paper. MasnouandMorel [13] recentlyextendedtheseideas,presentinga very inspiring generalvariationalformulationfor disocclusionanda particularpracticalalgorithmimplementingsomeof the ideasin this formulation. Thealgorithmperformsin-paintingby joining with geodesiccurvesthepointsof theisophotesarriving at theboundaryof theregion to beinpainted.As reportedby the authors,the regions to be inpaintedare limited to havingsimpletopology, e.g.,holesarenotallowed.3 In addition,theanglewith which the level lines arrive at the boundaryof the inpaintedregion is not (well) preserved: thealgorithmusesstraightlines tojoin equalgrayvaluepixels. Thesedrawbackswill beexemplifiedlaterin thispaper. Ontheotherhand,weshouldnotethatthis is theclosesttechniqueto oursandhasmotivatedin partandinspiredourwork.

2.1 Our contribution

Algorithmsdevisedfor film restorationarenot appropriatefor ourapplicationsincethey normally work on relatively small regionsandrely on theexistenceof informationfrom severalframes.

On theotherhand,algorithmsbasedon texturesynthesiscanfilllarge regions, but requirethe userto specify what texture to putwhere.This is a significantlimitation of theseapproaches,asmaybeseenin examplespresentedlater in this paper, wheretheregionto beinpaintedis surroundedby hundredsof differentbackgrounds,someof thembeingstructureandnot texture.

The techniquewe proposedoesnot requireany userinterven-tion, oncethe region to be inpaintedhasbeenselected.Thealgo-rithm is ableto simultaneouslyfill regionssurroundedby differentbackgrounds,without theuserspecifying“what to put where.” Noassumptionson the topologyof the region to be inpainted,or onthe simplicity of the image,are made. The algorithm is devisedfor inpaintingin structuredregions(e.g.,regionscrossingthroughboundaries),thoughit is not devised to reproducelarge texturedareas.As we will discusslater, the combinationof our proposedapproachwith texturesynthesistechniquesis thesubjectof currentresearch.

3 The digital inpainting algorithm

3.1 Fundamentals

Let � standfor theregion to beinpainted,and ��� for its boundary(noteonceagainthatnoassumptiononthetopologyof � is made).

2Sincetheregionto beinpaintedcanbeconsideredasoccludingobjects,removing occlusionsis analogousto imageinpainting.

3This is not intrinsic to thegeneralvariationalformulationthey propose,only to thespecificdiscreteimplementationthey perform.

Figure1: 1D signals���� �� and ���� �� .Intuitively, thetechniqueweproposewill prolongtheisophotelinesarriving at ��� , while maintainingthe angleof “arrival.” We pro-ceeddrawing from ��� inward in this way, while curving the pro-longationlines progressively to prevent themfrom crossingeachother.4

Beforepresentingthe detaileddescriptionof this technique,letus analyzehow expertsinpaint. Conservatorsof the MinneapolisInstituteof Arts wereconsultedfor this work andmadeit cleartousthatinpaintingis a very subjective procedure,differentfor eachwork of art and for eachprofessional.Thereis no suchthing as“the” way to solve theproblem,but theunderlyingmethodologyisasfollows: (1.) Theglobalpicturedetermineshow to fill in thegap,thepurposeof inpaintingbeingto restoretheunity of thework; (2.)Thestructureof theareasurrounding� is continuedinto thegap,contourlines are drawn via the prolongationof thosearriving at��� ; (3.) Thedifferentregionsinside � , asdefinedby thecontourlines,arefilled with color, matchingthoseof ��� ; and(4.) Thesmalldetailsarepainted(e.g.little whitespotsonanotherwiseuniformlybluesky). In otherwords,“texture” is added.

Our algorithmsimultaneouslyanditeratively performsthesteps(2.) and(3.) above. (In thediscussionsectionwe will arguehowboth stepscanbe performedseparately, andwe will alsodiscussstep(4.).) We progressively “shrink” thegap � by prolongingin-ward,in asmoothway, thelinesarriving at thegapboundary��� .

3.2 A one dimensional illustration

A simpleonedimensional(1D) exampleis first usedto illustrateour technique.This is doneonly for a didacticpurpose,sinceouralgorithmwasdevisedfor 2D, andthereareothertechniques(suchassplines)that might yield betterinterpolationresultsin 1D. Our1D examplewill helpto explainhow wetranslatethebasicinpaint-ing ideaspresentedabove into aworkingalgorithm.Considera1D(discrete)signal � �� ������ ����������� ( � �!�"�#�%$&� ' ) anda “box-function” masksignal �(�� �� , as illustratedin Figure1. The region� to inpaint is given by the setof pointswhere �(�� )�+*, � , whileits boundary��� arethepointsQ andP. Let -� ' bethedirectionofincreasing .

The progressive natureof the inpaintingproceduresuggestsan“evolution” approach.We definea family of 1D signals �.�� /�102�3�� �!�"�#�!45� ����67�8�9� ( � �!���#�:$;� '<�=� ����6>�2$;� ' ) where0 is the“time” parameter, suchthat �.�� /�?��� , �=@�� )� and AB DCFE�GIHJ�.�� /�/02� ,�LK%�� �� , where�LKM�� �� is theoutputof thealgorithm(inpaintedimage).At eachiterationstep0 wewantto construct�.�� /�/02� given �.�� ?�)0 -N � .

For the1D case,the“prolongationof linesinto � ” is interpretedas“the continuationof the function ���� �� from the left of Q to itsright,andfromtherightof Pto its left, in asmoothway.” Therefore,

4Thistypeof informationpropagationis relatedandmightbeapplicableto velocityfieldsextensionin level-settechniques[14].

Q R

a

b

c

Figure2: Discreteexample;seetext..

in orderto inpaintin thedirectionof -� ' , thefollowing pseudo-coderealizestheideasabove:for OQPFR to OQPTSVU�WQX1Y/Z%[]\_^`[Ja_bcY/Z=d=bDeB\�O!f

for eachpoint (pixel) gBeih�jQkcomputethesmoothnessof a=lmgieBh : n�logieBh2p PFa=lq�q gieBhcomputethesmoothnessof a=lmgie![sr"h : n�lmgBe�[3r?h�p PTa=lq�q giet[sr"hcomputethesmoothnessvariationin [ [u S :a=lv gieBh�p PTn�logieBh.[]n�logiet[sr"ha=l�wox=gieih.PTa=logieBh!y{z|bDa_lv gBeih

end

end,

wherethe index denotesspatialposition, the superindex 0 de-notesthe artificial evolution time ( �.�� /�/02� , � E �� )� ), the subindex} denotes(discretized)spatialdifferentiationin thehorizontalaxisdirection, and ~I� controlsthe velocity of the evolution. We arechangingin timethefunction �.�� /�/02� sothatateachpoint , � E@��� �� ��tendsto follow � E �� - N � . Thepurposeof thesmoothnessestimationis to ensurecontinuitynotonly of �.�� /�102� but alsoof its derivatives.Our discretesmoothnessestimatoris the secondspatialderivativeof �.�� /�/02� , � E �� )� , � E�� �� �� , which in discreteform canbewrittenas � E �� �� , � E �� - N � ->� � E �� ������ E �� .� N � .To explain why this works, considerFigure2. The continuousline denotesthepartof �.�� ?�10:� thatis notchangedby thealgorithmsinceit is outside� . To theright of Q, thethreedashedlinesdenotethreepossibleinitial conditionsfor �.� } �1�@� inside � . Beingstraightlines, �`�D ��?��� , � in the threecases.At thepoint Q on theotherhand,thevalueof ���D�]�?��� is differentin eachcase:

If a(gi�M��R=hQP�d , n�gi�`�cR=hJ��R , a v gi�M�cR=h�P�n�gi�M�cR=h�[�n�gB���DR=h]��R , anda(gi�M�cR=h decreases;

If a(gi�M�cR=h.P�X , n�gB���cR=h.P<R , a v gi�M�)R_h.PTn�gB�M�DR_h_[%n�gi���DR=h.PFR , and a(gi�M�cR=hdoesnotchange;

If a(gi�M��R=h.PT� , n�gB���cR=h:�3R , a v gi�M�)R_h.PFn�gB�M�cR_h�[�n�gB���cR=h:�{R , and a(gi�M�cR=hincreases.

In all threecasestheevolutionis deforming�.�� /�102� inside � intoacontinuationof ���� �� to theleft of Q.

Figure3 shows threestagesof theevolutionof thesignalin Fig-ure1. Thedottedline correspondsto the initial condition, �.�� /�/��� .Thedashedline correspondsto an intermediatestate.Thecontin-uousline correspondsto �LKM�� �� , the steadystateof the evolution.Noticehow continuityup to thefirst derivative is obtainedfor thepoint Q. That is not thecasefor P, sincethedirectionof propaga-tion is -� ' andnot - -� ' . Wewill laterseethatthis is not a problemfor the2D case.We have tested1D exampleswith differentinitialconditions,andfound that the algorithmalwaysconvergesto thesamesolutionwithin a0.1%errormargin. Theformalstudyof thisexperimentalresultis of theoreticalinterest(seeSection5).

3.3 The 2D inpainting algorithm

Let us re-write the 1D discreteequationdescribingthe rate ofchangeof �.�� /�/02� : � E@��� �� �� , � E �� ����7~I�)� E� �� ���� (1)

Figure3: Threestagesof theevolutionof �.�� /�102� in Figure1.

Figure4: Propagationdirectionasthenormalto thesigneddistanceto theboundaryof theregion to beinpainted.

where � E� �� )� , � E �� �� - � E �� - N ���5�. ��F��� (2)� E �� �� , � E �� - N � ->� � E �� ������ E �� .� N ��� (3)

In otherwords,we estimatethesmoothness� E �� )� of our func-tion andcomputeits changealongthe - -� ' direction,adaptingtheimagein the region to be inpaintedwith this variationof smooth-nessquantity. Let usnow seehow to extendtheseideasinto 2D.

Let ���� /�����3��� �������|4;� �!�"'{��� � , with � �������%4¡� ���?'5�Q$� '¢4£� ' , be a discrete2D gray level image. The inpaintingpro-cedurewill constructa family of images �.�� /�D���?02���£� �!�"�#�Q4� �!�?'{�54¤� �!��6;�¥� � such that �.�� /�c���?�@� , ���� /�c��� andAB DC E�G�H �.�� ?�D�@�10:� , �LKM�� /�c��� , where �LKM�� /�c��� is the outputof thealgorithm(inpaintedimage).

Following the 1D procedure,first we estimatethe smoothness� E �� /����� of our function. For this purposewe may usea simplediscreteimplementationof theLaplacian: � E �� /�c����� , � E�� �� /�c������ E¦=¦ �� /�c��� . Othersmoothnessestimatorsmight beused,thoughsat-isfactoryresultswerealreadyobtainedwith this very simpleselec-tion.

Then,we mustcomputethechangeof this valuealong - -� ' . Inorderto do this we mustfirst definewhat the direction -� ' for the2D informationpropagationwill be.Onepossibilityis to define -� 'asthenormalto thesigneddistanceto ��� , i.e.,at eachpoint �� /�c���in � thevector -� ';�� ?�D��� will benormalto the“shrinkedversion”of��� to which �� ?�D��� belongs,seeFigure4. This choiceis motivatedby thebelief thatapropagationnormalto theboundarywould leadto the continuity of the isophotesat the boundary. Instead,whathappensis thatthelinesarriving at ��� curve in orderto align with-� ' , seeFigure5. This is of coursenot whatwe expect. Note thattheorientationof ��� is not intrinsic to the imagegeometry, sincetheregion to beinpainted! ! is arbitrary.

If isophotestendto align with -� ' , thebestchoicefor -� ' is thenthe isophotesdirections. This is a bootstrappingproblem: hav-ing the isophotesdirectionsinside � is equivalent to having theinpaintedimageitself, sincewe caneasily recover the gray levelimagefrom its isophotedirectionfield ( seethediscussionsectionand[15]).

Figure5: Unsuccessfulchoiceof the informationpropagationdi-rection.Left: detailof theoriginal image,region to beinpaintedisin white. Right: restoration.

Weusethena timevaryingestimationof theisophotesdirectionfield: for any given point �� ?�D��� , the (discretized)gradientvector§ � E �� /�c��� givesthe directionof largest(spatial)change,while its90 degreesrotation

§]¨ � E �� /�c��� is thedirectionof smallest(spatial)change,sothevector

§ ¨ � E �� /�c��� givestheisophotesdirection.Ourfield -� ' is thengivenby thetime-varying -� '7�� /�����/02� , §J¨ � E �� /�c��� .We areusinga time-varyingestimationthat is coarseat thebegin-ning but progressively achieves the desiredcontinuity at ��� , in-steadof a fixed field -� ';�� ?����� that would imply to know thedirec-tionsof theisophotesfrom thestart.

Note that the field is not normalizedas it was in the 1D case,where -� ' wasjust thedirectionof increasing . Thereasonfor thischoicereliesonthenumericalstabilityof thealgorithm,andwill bediscussedin thefollowing subsection.

Sincewe areperforminganinpaintingalongthe isophotes,it isirrelevantif

§ ¨ � E �� /����� is obtainedasaclockwiseor counterclock-wise rotationof

§ � E �� /�c��� . In both cases,the changeof � E �� /�c���alongthosedirectionsshouldbeminimum.

Recapping,we estimatea variationof thesmoothness,givenbya discretizationof the 2D Laplacianin our case,andproject thisvariation into the isophotesdirection. This projectionis usedtoupdatethevalueof theimageinsidetheregion to beinpainted.

To ensurea correctevolution of the directionfield, a diffusionprocessis interleavedwith the imageinpaintingprocessdescribedabove. That is, every few steps(seebelow), we applya few itera-tionsof imagediffusion.Thisdiffusioncorrespondsto theperiodi-cal curvingof linesto avoid themfrom crossingeachother, aswasmentionedin section3.1. We useanisotropicdiffusion, [16, 17],in orderto achieve this goalwithout losingsharpnessin therecon-struction.In particular, weapplyastraightforwarddiscretizationofthe following continuous-time/continuous-spaceanisotropicdiffu-sionequation:�.��m� � } �/©.�1�/� ,#ª�« � } �1©o�1¬2� } �1©.�1�/��­ § �.� } �)©.�1�?�=­������ } �)©o�M�<� « (4)

where� « is adilationof � with aball of radius® , ¬ is theEuclideancurvatureof theisophotesof ¯ and ª « � } �1©t� is asmoothfunctionin� « suchthat ª@« � } �1©t� , � in � «�° � , and ª�« � } �/©t� , N at thesetofpointsof � whosedistanceto ��� is larger that ® (this is a way toimposeDirichlet boundaryconditionsfor theequation(4)).

3.4 Discrete scheme and implementation details

The only input to our algorithmarethe imageto be restoredandthemaskthatdelimitstheportionto beinpainted.As apreprocess-ing step,thewholeoriginal imageundergoesanisotropicdiffusionsmoothing.The purposeis to minimize the influenceof noiseontheestimationof thedirectionof theisophotesarrivingat ��� . Afterthis, theimageenterstheinpaintingloop,whereonly thevaluesin-side � aremodified.Thesevalueschangeaccordingto thediscreteimplementationof the inpaintingprocedure,which we proceedto

describe.Every few iterations,astepof anisotropicdiffusionis ap-plied (a straightforward,centraldifferencesimplementationof (4)is used;for detailssee[16, 17]). This processis repeateduntil asteadystateis achieved.

Let � E �� ?�D��� standfor eachone of the imagepixels inside theregion � at the inpainting“time” 0 . Then,the discreteinpaintingequationborrowsfromthenumericalanalysisliteratureandis givenby � E@��� �� /�c��� , � E �� /�c���2�7~I�)� E� �� ?������� (5)

where

� E� �� ?�D��� , ± -�- �² � E �� ?������³ -� '7�� /�c���?02�´´ -� '7�� /�c���?02� ´´)µ ­ § � E �� /�����=­�� (6)

-c- �² � E �� /�����|� , �D� E �� �� N ����� - � E �� - N �D�����"� E �� ?�D��� N � - � E �� /�c� - N �/���(7)� E �� /�c��� , � E�� �� /��������� E¦_¦ �� /������� (8)-� ';�� ?�D�@�?02�´´ -� ';�� ?�D�@�?02� ´´ � , � - � E¦ �� /�c�����?� E� �� ?�D���/�¶ ��� E� �� /�c���/�/·��&��� E¦ �� /�c���/�/·%��® � (9)

¸ E �� /����� , -c- �² � E �� /�c���8³ -� ';�� ?�D�@�?02�´´ -� ';�� ?�D�@�?02� ´´ � (10)

and

­ § � E �� ?�D���=­ ,º¹»¼ »½¶ ��� E�¾c¿V� · ����� E�LÀLÁ{� · ����� E¦=¾c¿�� · �&��� E¦ÀLÁs� · �when

¸ E{ �¶ ��� E�¾�Á �/·8����� E�LÀ¿ �/·������ E¦=¾�Á �/·��#��� E¦À¿ �/·(�when

¸ E{Ã �(11)

Wefirst computethe2D smoothnessestimation� in (8) andtheisophotedirection -� 'ÅÄ ´´ -� ' ´´ in (9). Then in (10) we compute E ,the projectionof - �² � onto the (normalized)vector -� ' , that is, wecomputethe changeof � along the direction of -� ' . Finally, wemultiply

¸ E by a slope-limitedversionof thenormof thegradientof theimage, ­ § ��­ , in (11). A centraldifferencesrealizationwouldturn the schemeunstable,and that is the reasonfor usingslope-limiters. The subindexes � and Æ denotebackward and forwarddifferencesrespectively, while thesubindexes C and � denotetheminimum or maximum,respectively, betweenthe derivative andzero(we have omittedthespacecoordinates�� ?�D��� for simplicity);see[18] for details. Finally, let us note that the choiceof

§]¨ �insteadof anormalizedversionof it (aswasthecasein 1D) allowsfor asimplerandmorestablenumericalscheme;see[19, 20].

Whenapplyingequations(5)-(11)to thepixelsin theborder���of the region � to be inpainted,known pixels from outsidethisregion areused.That is, conceptually, we computeequations(5)-(11)in theregion � « (an ® dilationof � ), althoughupdatethevaluesonly inside � (thatis, (5) is appliedonly inside � ). Theinformationin thenarrow band� « - � (isophotedirectionsandgrayvalues)ispropagatedinside � . Propagationof this information,both gray-valuesandisophotesdirections,is fundamentalfor the successofthealgorithm.

In the restorationloop we perform Ç stepsof inpaintingwith(5), then È stepsof diffusion with (4), again Ç stepsof (5), andso on. The total numberof stepsis É . This numbermay be pre-established,or thealgorithmmaystopwhenchangesin the image

Figure6: Relationbetweenthe �D I�"Ê��?È�� color modelandtheoneusedin thisarticle,�ÌË.�?Í= D0�Î��"Í= c0�Ï|� .arebelow a giventhreshold.Thevalueswe useare: Ç , NÐ �"È ,� ��~�� , �!� N . The valueof É dependson the sizeof � . If � isof considerablesize,amultiresolutionapproachis usedto speed-uptheprocess.

Color imagesare consideredas a setof threeimages,and theabove describedtechniqueis appliedindependentlyto eachone.To avoid the appearanceof spuriouscolors,we usea color modelwhich is very similar to theLUV model,with oneluminanceandtwo chromacomponents.SeeFigure6.

4 Results

TheCPUtime requiredfor inpaintingdependson thesizeof � . Inall the color examplesherepresented,the inpaintingprocesswascompletedin lessthan5 minutes(for thethreecolor planes),usingnon-optimizedC++coderunningonaPentiumIIPC(128MbRAM,300MHz)underLinux. All theexamplesuseimagesavailablefrompublicdatabasesover theInternet.

Figure7 shows, on the left, a syntheticimagewith the regionto inpaint in white. Here � is large (30 pixels in diameter)andcontainsahole.Theinpaintedreconstructionis shown ontheright.Notice that contoursare recovered,joining points from the innerand outer boundaries.Also, thesereconstructedcontoursfollowsmoothlythedirectionof theisophotesarriving at ��� .

Figure8 shows a deterioratedB&W imageand its reconstruc-tion. As in all the examplesin this article, the useronly suppliedthe“mask” image,shown in Figure9. This imagewasdrawn man-ually, usinga paintbrush-like program. The variableswereset tothe valuesspecifiedin the previous section,andthe numberof it-erationsÉ wassetto 3000. Whenmultiresolutionis not used,theCPUtime requiredby theinpaintingprocedurewasapproximately7 minutes.With a 2-level multiresolutionscheme,only 2 minuteswereneeded.Observe thatdetailsin thenoseandright eye of themiddlegirl couldnot becompletelyrestored.This is in partduetothefact that themaskcoversmostof therelevant information,andthereis not muchto bedonewithout theuseof high level prior in-formation(e.g.,thefactthatit is aneye). Theseminorerrorscanbecorrectedby themanualproceduresmentionedin theintroduction,andstill theoverall inpaintingtime would bereducedby ordersofmagnitude.In Figure10, top,we show a differentinitial conditioninside � . The inpaintedimage(bottom)is practicallyidenticaltothe one in Figure9, therebyshowing the robustnessof the algo-rithm.

Figure11showsavandalizedimageandits restoration,followedby anexamplewhereoverimposedtext is removedfrom theimage.Thesearetypical exampleswheretexture synthesisalgorithmsasthosedescribedin theintroductioncannot beused,sincethenum-berof differentregionsto befilled is very large.

Figure7: Syntheticexample: � is shown in white. Topology isnot an issue,and the recovered contourssmoothlycontinuetheisophotes.

Figure12shows theprogressive natureof thealgorithm:severalintermediatestepsof theinpaintingprocedureareshown for adetailof figure11.

Finally, Figure 13 shows an entertainmentapplication Thebungeecordandtheknot tying theman’s legshave beenremoved.Given the size of � a 2-level multiresolutionschemewas used.Hereit becomesapparentthat it is the userwho hasto supplythealgorithmwith themaskingimage,sincethechoiceof theregiontoinpaintis completelysubjective.

5 Conclusions and future work

In this paperwe have introduceda novel algorithmfor imagein-paintingthatattemptsto replicatethebasictechniquesusedby pro-fessionalrestorators.Thebasicideais to smoothlypropagateinfor-mationfrom thesurroundingareasin the isophotesdirection. Theuserneedsonly to provide the region to be inpainted,the rest isautomaticallyperformedby the algorithmin a few minutes. Theinpaintedimagesaresharpandwithout color artifacts.Theexam-ples shown suggesta wide rangeof applicationslike restorationof old photographsand damagedfilm, removal of superimposedtext, andremoval of objects. Theresultscaneitherbe adoptedasa final restorationor beusedto provide aninitial point for manualrestoration,therebyreducingthetotal restorationtimeby ordersofmagnitude.

Oneof themainproblemswith ourtechniqueis thereproductionof largetexturedregions,ascanbeseenin Figure14. Thealgorithmhereproposedis currentlybeingtestedin conjunctionwith texturesynthesisideasto addressthis issue.

The inpainting algorithm herepresentedhasbeenclearly mo-tivatedby and hasborrowed from the intensive work on the useof Partial DifferentialEquations(PDE’s) in imageprocessingandcomputervision. When“blindly” letting thegrid go to zero,thein-paintingtechniquein equations(5)-(11),naively resemblesa thirdorder equation(not resultingfrom a variationalgradientdescentflow, seebelow). Althoughaddingregularizationtermsmightmakethis high orderequationstableandallow a formal analysis,5 to thebestof our knowledgethecompleteunderstandingof suchtypeofequationsis not just beyondthescopeof this paperbut beyondthecurrentstateof mathematicalknowledge(althoughresultsfor otherhighorderequations,whichmightberelevantfor imageprocessingaswell, areavailable,e.g.,[22]). In addition,thediscretealgorithmhereintroducedusesas“boundaryconditions”bothgrayvaluesandisophotedirectionspresentat thenarrow bandsurroundingthe re-gion to beinpainted.This is done,asexplainedabove, performingthecomputationsin thedilatedregion � « . All this importantinfor-mationneedsto beincorporatedto suchanequationas“extra” (and

5Note thatnumericalimplementationsof PDE’s intrinsically regularizetheequations.

fundamental)boundaryconditions.Nevertheless,this suggeststheinvestigationof theuseof lower, secondorder, PDE’sto addresstheinpaintingproblem.Moreover, having theseequationsto begradi-entdescentflows of variationalformulationspermitsthe inclusionof the narrow bandinformationsimply by changingthe limits ofintegration. Onepossibleapproachto follow is to first reconstructtheisophotedirectionsÑ (a planarvectorfield), andfrom themthecorrespondinggrayvalues� . Oncethe isophotedirectionsarere-constructed,thegradientdirectionis given,andwecanreconstructthe gray levels by finding an imageconsistentwith thesegradi-entdirections.Althoughthis canbedoneusinga onedimensionaltransportequation,the existenttheorylimits the possiblegradientfields,therebydisqualifyingedges.We cantake thena variationalapproachandfind theimage� minimizing Ò�ÓoÔ(�?­ § ��­ - Ñ`³ § �!�)Õ�� ,where Ñ is the reconstructedgradientdirection inside � « (an ad-ditional term that penalizesthe deviation from the real imageinthe narrow band � « - � canbe addedaswell). The minimal �is obtainedvia gradientdescentflows (secondorderPDE).There-fore,whatis left is to reconstructthevectorfield Ñ . This canagainbe achieved usinga variationalformulationon Ñ with the properboundaryconditionsto guaranteethecontinuityof thevectorfieldacross��� (seealso [13, 21]). Both variationalformulationscanactuallybecoupledto simultaneouslyrecover theisophotes(vectorfield) andgrayvaluesvia thecorrespondinggradientdescentflow,obtainingasetof coupledsecondorderPDE’s.6 Preliminaryresultsin this directionarepromisingandwill befurther investigatedandreportedelsewhere[23]. Oneof theadvantagesof this approachisthat formal resultsregardinguniquenessandexistenceof thesolu-tions canbe shown, therebyreducingthe necessityto rely mainlyonexperimentalvalidationsasdoneuntil now with thebasicimageinpaintingalgorithmsreportedin theliterature.

Acknowledgments

Thiswork startedwhentheauthorswerevisiting theInstituteHenriPoincarein Paris, France. We thank the organizersof the quar-ter on “MathematicalQuestionsin SignalandImageProcessing”andthe Institutefor their hospitalityandpartial financialsupport.We would like to thankTom RobbinsandElizabethBuschorfromtheUpperMidwestConservationAssociationfor theirhelpin link-ing our work with actualart restorationandconservation. We alsothank Dr. SantiagoBetelu, University of Minnesota,Prof. StanOsher, UCLA, andProf. EeroSimoncelli,NYU, for very interest-ing discussionsandfeedback.This work waspartially supportedby a grant from the Office of Naval ResearchONR-N00014-97-1-0509,the Office of Naval ResearchYoungInvestigatorAward,the PresidentialEarly CareerAwardsfor ScientistsandEngineers(PECASE),a NationalScienceFoundationCAREERAward, andby the NationalScienceFoundationLearningandIntelligentSys-temsProgram(LIS).

References

[1] S.Walden.TheRavishedImage. St.Martin’sPress,New York,1985.

[2] G. Emile-Male. TheRestorer’s Handbookof EaselPainting.VanNostrandReinhold,New York, 1976.

[3] D. King. TheCommissarVanishes. HenryHolt andCompany,1997.

6Intuitively, it is clear that we cannot addressthe inpaintingproblemwith a singlesecondorderPDE,sinceregularizationconstraintson vectorfieldsareneeded(in [13], thegeneralvariationalformulationsuggestedwillnormallyleadto a fourthordergradientdescentflow).

[4] L. Joyeux,O. Buisson,B. Besserer, S. Boukir. DetectionandRemoval of Line Scratchesin motionPicture Films. Proceed-ings of CVPR’99, IEEE Int. Conf. on ComputerVision andPatternRecognition,Fort Collins,Colorado,USA, June1999.

[5] A.C. Kokaram,R.D. Morris, W.J. Fitzgerald,P.J.W. Rayner.Detectionof missingdata in image sequences. IEEE Transac-tionson ImageProcessing11(4),1496-1508,1995.

[6] A.C. Kokaram,R.D. Morris, W.J. Fitzgerald,P.J.W. Rayner.Interpolationof missingdatain imagesequences. IEEETrans-actionson ImageProcessing11(4),1509-1519,1995.

[7] C. Braverman. Photoshopretouching handbook. IDG BooksWorlwide,1998.

[8] A. Hirani andT. Totsuka. CombiningFrequencyand spatialdomaininformationfor fast interactive image noiseremoval.ComputerGraphics,pp.269-276,SIGGRAPH96,1996.

[9] A. Efros andT. Leung,“Texture synthesisby non-parametricsampling,” Proc.IEEEInternationalConferenceComputerVi-sion, pp.1033-1038,Corfu,Greece,September1999.

[10] D. Heeger and J. Bergen. Pyramid basedtexture analy-sis/synthesis. ComputerGraphics,pp. 229-238,SIGGRAPH95,1995.

[11] E. Simoncelli and J. Portilla. Texture characterizationviajoint statisticsof waveletcoefficientmagnitudes. 5thIEEEInt’ lConf.on ImageProcessing,Chicago,IL. Oct4-7,1998.

[12] M. Nitzberg, D. Mumford, andT. Shiota,Filtering, Segmen-tation,andDepth, Springer-Verlag,Berlin, 1993.

[13] S. MasnouandJ.M. Morel. Level-linesbaseddisocclusion.5th IEEE Int’ l Conf. on ImageProcessing,Chicago,IL. Oct4-7,1998.

[14] S.Osher, personalcommunication,October1999.

[15] C. Kenney andJ.Langan.A new image processingprimitive:reconstructingimagesfrommodifiedflow fields. UniversityofCaliforniaSantaBarbaraPreprint,1999.

[16] P. PeronaandJ.Malik Scale-spaceandedge detectionusinganisotropicdiffusion. IEEE-PAMI 12,pp.629-639,1990.

[17] L. Alvarez,P.L. Lions,J.M.Morel. Imageselectivesmoothingand edge detectionby nonlineardiffusion. SIAM J. Numer.Anal. 29,pp.845-866,1992.

[18] S. OsherandJ. Sethian. Frontspropagating with curvaturedependentspeed: algorithmsbasedon Hamilton-Jacobi for-mulations. Journalof ComputationalPhysics,79:12-49,1988.

[19] A. MarquinaandS.Osher. Explicit algorithmsfor a new timedependentmodelbasedon level setmotionfor nonlinearde-bluring andnoiseremoval. UCLA CAM Report99-5,January1999.

[20] L. Rudin,S. OsherandE. Fatemi. Nonlineartotal variationbasednoiseremoval algorithms. PhysicaD, 60, pp. 259-268,1992.

[21] L. I. RudinandS.Osher, Total variationbasedimagerestora-tion with freelocal constraints. Proc.IEEE-ICIPI, pp. 31-35,Austin,Texas,1994.

[22] A. Bertozzi Themathematicsof moving contactlines in thinliquid films. NoticesAmer. Math.Soc.,Volume45,Number6,pp.689-697,June/July1998.

[23] C. Ballester, M. Bertalmio,V. Caselles,andG. Sapiro,“Inter-polationof planarvectorfields andapplications,” in prepara-tion.

Figure8: Restorationof anold photograph.

Figure9: Theuserdefinestheregion to inpaint(hereshown in red).

Figure10: Fromadifferentinitial conditioninside � wearriveat averysimilar result..

Figure11: Restorationof acolor imageandremoval of superimposedtext.

Figure12: Progressivenatureof thealgorithm:severalintermediatestepsof thereconstructionof figure11(detail).

Figure13: Thebungeecordandtheknot tying theman’s feethavebeenremoved.

Figure14: Limitationsof thealgorithm:textureis not reproduced.