1

Parameter Optimisation for Texture CompletionHoang M. Nguyen, Burkhard C. Wunsche, Patrice Delmas, and Christof Lutteroth

Dept of Computer Science, University of Auckland, Auckland, New Zealandjustin.nguyen@auckland.ac.nz {burkhard, p.delmas, lutteroth}@cs.auckland.ac.nz

Abstract—In 3D reconstruction applications frequently large mesh sections are missing. While the geometry can becreated with mesh completion techniques, reconstructing the texture is more difficult. An exemplar-based synthesis usuallyrequires that the missing region contains only one texture pattern. Image inpainting techniques work for more complextextures, but were designed primarily to fill narrow missing regions and tend to produce undesirable results when themissing region is large. We solve this problem by exploiting the fact that most 3D objects contain different surface regionswith similar textures. We use an appearance space to identify suitable texture patches fitting with the boundary of themissing region, and fill the region by using a patch-based synthesis combined with Poisson interpolation. We evaluate theeffect of different parameters and demonstrate its improved performance over existing inpainting techniques.

Keywords—texture retouching, texture inpainting, texture completion, image information recovery

F

1 INTRODUCTION

Over the past couple of years 3D reconstruc-tion technologies have improved tremendouslyand are now available in the consumer-leveldomain. Examples include structured lighting(Kinect), laser scanners, and image-based mod-eling. In many applications the resulting meshcontains large missing regions, e.g. because thesurface was invisible to the utilised sensor. Sur-face reconstruction techniques, such as PoissonSurface Reconstruction, and mesh completiontechniques can create a watertight surface, butare unable to reconstruct texture information.

Missing textures can be obtained with texturesynthesis methods, but these usually employ asingle exemplar and assume a consistent tex-ture over the missing region. Image inpaintingtechniques work for more complex textures, butwere designed primarily to fill narrow missingregions and tend to produce undesirable resultswhen the missing region is large.

3D objects in the real-world usually containrepeating geometries and textures, e.g. the sidesof an animal, components of a plant (stem,branch, blossom), or man-made objects whichare usually designed using symmetries andself-similarities (e.g. walls and windows of ahouse).

In this paper we perform texture completionby exploiting the fact that most 3D objects

contain different surface regions with similartextures. We use an appearance space to iden-tify suitable texture patches fitting with theboundary of the missing regions, and fill theregion by using a patch-based synthesis com-bined with Poisson interpolation.

The remainder of this paper is organisedas follows: After a brief discussion of existingimage inpainting techniques in section 2, wedescribe our inpainting algorithm in section 3.Section 4 presents results and section 5 con-cludes this paper.

2 RELATED WORK

Image in-painting techniques can be dividedinto two classes. Pixel-based methods fill themissing region pixel-by-pixel starting from theboundary, whereas patch-based methods usu-ally add entire patches and minimise disconti-nuities between texture regions.

The arguably best known and most suc-cessful algorithm amongst pixel-based inpaint-ing methods was proposed by Bertalmio et al.[1]. The authors attempt to replicate manualinpainting by propagating the known colorvalues into a missing region along so calledisophotes, representing the smallest spatialchange of color values and structures.

Drori et al. [2] use adaptive circular fragmentsto operate on different scales to capture both

2

global and local structures and approximate themissing region.

Telea et al. [6] present an inpainting techniquebased on a fast marching method for level setapplications. The method is simple and con-siderably more efficient than other pixel-basedmethods.

Pixel-based methods often fail to properly re-construct larger structures with semantic mean-ing, e.g. leaves. Criminisi et al. [3] iterativelyselect a “best-fit” rectangular patch and copy itover to the target region. The order in whichboundary pixels of the missing region are pro-cessed is based on the amount of informationavailable for that pixel and whether it has anyprominent features.

Cheng et al. [4] update the priority equationof [3] and made it adjustable to the structuraland textural information specific to an image.Ignacio et al. [5] extend the concept of Cri-minisi’s method and apply it in the waveletdomain.

3 ALGORITHM

We employ a similar patch search and insertionconcept as Criminisi et al., but use appearancespace attributes and principal component anal-ysis to improve region matching quality andspeed. In addition, our method smoothly fusespatches together to remove all visible seams.

3.1 Candidate Patch IdentificationThe algorithm’s performance is significantlyeffected by the ability to identify the patch inthe image that retains the highest resemblanceto the processed patch. This is achieved byiteratively traversing through each pixel of theimage outside the missing region and comput-ing the similarity of the patch centered aroundthat pixel and the original patch. Instead ofusing the standard Sum of Squared Differences(SSD) to measure the similarity of two givenpatches, we employ appearance space attributes,which provide much more information andthus improve the search result.

When searching for a matching patch, weconsider for each pixel an 11 × 11 pixel neigh-bourhood. For each pixel of this neighborhood

we consider RGB colours, the gradient vector,as well as the signed Euclidean distance to theclosest dominant feature in the original texture.

Gradient Vector: We estimate the gradientof an intensity image I, ( ∂I

∂x, ∂I∂y

) by using theconvolution kernels of a standard Sobel opera-tor. The two kernels for the x and y directionsare:

Sx = (1 2 1)T · (1 0 − 1) (1)

Sy = (1 0 − 1)T · (1 2 1) (2)

The attribute gradient image is then definedas:

A(p) = ((Sx ∗ I)(p), (Sy ∗ I)(p)) (3)

where p denotes an image pixel.

Signed Feature Distance: The signed fea-ture distance of a pixel p is defined as thedistance of p to the closest pixel q ∈ M forwhich M(p) 6= M(q) where M is a binary imagecreated by applying the canny edge detectionmethod [11] on the input image.

The distance between two pixels is definedusing the following equations (adapted from[9]):

A(p) = s · (maxr ∈ M

λ(r)− λ(p)) (4)

λ(x) =1

|qx − x|(5)

f(x) =

{1 if M(p) = 1

−1 if M(p) = 0(6)

The vector (qx−x) points from x to the closestpixel q in M.

The entire attribute information is encapsu-lated into a 11 × 11 × (3 + 2 + 1) = 726-dimensional vector. Determining the similarityof two given patches by comparing two 726-dimensional vectors is not efficient. In order tomake the appearance space more practicable,the 726 dimensional vectors are projected intolow-dimensional vectors using principal compo-nent analysis (PCA) ([9], [10]). In our method,the dimensionality is reduced to 8, which inour experiments with different image typesproduced the best results.

3

The clear advantage of the attribute spaceover the conventional SSD is that the attributespace approach permits any meaningful infor-mation about the pixels and their surroundingto be embedded for matching purposes. Byreducing the dimensionality, the computationtime can be kept manageable.

3.2 Patch FusionThe final step is to replicate the content of thecandidate patch and smoothly blend it withthe target region. We employ a Poisson-guidedinterpolation approach proposed by [7] for thistask.

The goal is to adjust the colour informationof patch ΨB, while preserving the relative infor-mation (image gradient) as much as possible, sothat the transition between the newly modifiedpatch ΨC and the rest of the image is gracefullyblended.

4 EVALUATION

In this section, we investigate the effect ofdifferent algorithm parameters and evaluate itsperformance over popular existing algorithms.

4.1 Effect of Parameters

Fig. 1. The original image is on the left, whilethe damaged image is on the right

Effect of appearance space attributes:Figure 2 shows an example in which the pre-vious damaged “lizard” image (Figure 1) isinpainted using various different appearancespace attributes. The following combinationsof appearance space attributes are used fortesting.• Case 1: RGB color, HSB color, signed

feature distance, horizontal and verticalgradient vectors.

Fig. 2. Appearance space attribute parameters:a) Color, HSB color, signed feature distance,horizontal and vertical gradient vectors. b) Color,signed feature distance, horizontal and verticalgradient vectors. c) Color and signed feature dis-tance. d) Color, horizontal and vertical gradientvectors.

• Case 2: RGB color, signed feature dis-tance, horizontal and vertical gradientvectors.

• Case 3: RGB color and signed featuredistance.

• Case 4: RGB color, horizontal and vertical

4

gradient vectors.As can be observed from figure 2, there is

little or no visual difference between the firstand second case. This indicates that adding theHSB color attribute does not result in findingbetter patches.

There is an obvious difference in the resultquality when the gradient vectors are removed.Some parts of the texture of the head becomefuzzy and appear incorrect. A small part of theline on the body of the lizard is missing.

The result quality further deteriorates whenremoving the signed feature distance attributein favor of the gradient vectors. The texturesbecome much more blurry and more missingfeatures are noted. This is probably due to thefact that inccorect patches have been identifiedand blended in this case.

We tested the different combinations of at-tributes with various images from different do-mains, and the results indicate that the optimalcombination of appearance space attributes isRGB color, signed feature distance, horizontal andvertical gradient vectors. Adding more attributesoften contributes little or no improvement andyet increases the computational cost.

Effect of attribute vectors’ dimensionality:In order to improve the efficiency, appear-ance attribute vectors are projected to a low-dimensional vectors before being compared toothers. In this section, we evaluate the effectsof the dimensionality of reduced appearanceattribute vectors on the the result quality.

Figure 3 shows an example in which appear-ance attribute vectors were reduced to differentdimensions (8, 7, 6, 5) before comparison. Re-ducing the original 726−dimensional attributevectors to a 8−dimensional vector yields goodresults. Missing textures are reconstructed welland seamlessly fused with the existing textures.We found that using more than 8 dimensionsdoes not result in a visible improvement of thetexture completion result.

Reducing the dimension further to 6 or 7increases blurriness. This is probably due to thefact that the reduced vectors did not containenough information for correct patches to befound.

When only 5 dimensions are retained, defor-mations start to appear due to the lack of infor-

Fig. 3. Appearance space attribute dimensions:a) 8−dimensional vectors. b) 7−dimensionalvectors. c) 6−dimensional vectors. d)5−dimensional vectors.

mation required for correct patch detections.Effect of Patch Size: Figure 4 demonstrates

the difference in the result quality when vary-ing the patch sizes. We found that in mostcases the ideal patch size is the range between7 and 11. Inpainted images in these cases areoften well-reconstructed. As the size of thepatches increases, the painted regions tend tobecome more blurry. This is probably because

5

the larger the patch size is, the more unrelatedfeatures from surrounding regions are involun-tarily taken into account leading to less accu-rate patch to be selected. Patch sizes smallerthan 7 increase the computational cost whilecontribute little improvement in term of thequality.

Fig. 4. Effect of patch sizes on the result quality.

4.2 Evaluation against Other InpaintingMethodsIn this section, we evaluate the performance ofour method against some of the best knownimage inpainting methods described in the lit-erature (Figure 5).

Bertalmio’s method [1] was not very success-ful in this test. It was able to reconstruct smallparts of the missing regions (near the bound-ary), but failed to interpolate textures further

for the middle regions resulting in patches withcolours not consistent with the regions neigh-borhood.

Telea’s method [6] outperformed Bertalmio’smethod both in terms of reconstruction qualityand efficiency. However, the inpainted regionsappear very blurry and unrealistic. This is ex-pected as pixel-based methods were designedto tackle only narrow missing regions.

Standard exemplar-based inpainting method(Criminisi [3]) performed reasonably well ex-cept for some artefacts. These artefacts arecaused by incorrect patches that were selectedas a result of employing the conventional SSD.

Our algorithm performed well in this testcase. Although the inpainted region still ex-hibits slight blurriness, the overall structureof the different scene components is correctlyrecovered.

5 CONCLUSION AND FUTURE WORK

We have presented a novel image inpaintingalgorithm for synthesizing large missing tex-ture regions from digital images. The results ofthis inpainting process is a new image in whichthe deterioration has been “inpainted” and re-verted in such a way that few visible tracesof it remain. The basic idea of our approachis to replicate missing textures by looking for“best-fit” texture patches in the source regionsand smoothly inserting these patches into themissing region to produce the final result.

Our solution offers two major improvementscompared to existing techniques. Patches forfilling in missing regions are found using anappearance space vector, which not only en-codes colour differences between regions, butalso colour gradients, feature distances andother measures for image similarity. The secondmajor improvement is the technique to combinethe patches filling in a missing region. We use aPoisson-guided interpolation to blend patchesto avoid visible seams.

We have evaluated our method’s perfor-mance against some of the best known inpaint-ing methods described in the literature andfound that our results are superior.

6

REFERENCES

[1] Marcelo Bertalmio and Guillermo Sapiro and VicentCaselles and Coloma Ballester, Image Inpainting, In Pro-ceeding SIGGRAPH ’00 Proceedings of the 27th annualconference on Computer graphics and interactive tech-niques, pp. 417–424

[2] Iddo Drori and Daniel Cohen-Or and Hezy Yeshurun,Fragment-Based Image Completion, ACM Transactions onGraphics, Vol. 22, pp. 303–312

[3] A. Criminisi and P. Perez and K. Toyama, Object re-moval by exemplar-based inpainting, ACM Transactionson Graphics, pp. 721–728

[4] Wen-Huang Cheng and Chun-Wei Hsieh and Sheng-KaiLin and Chia-Wei Wang and Ja-Ling Wu, Robust algorithmfor exemplar-based image inpainting, In Proceedings ofCGIV, pp. 64–69

[5] Ubirate Ignecio and Cleudio R Jung, Block-based imageinpainting in the wavelet domain, Visual Computing,pp. 733–741

[6] Alexandru Telea, An image inpainting technique based onthe fast marching method, Journal of Graphics Tool, Vol. 9,pp. 23–34

[7] Patrick Perez and Michel Gangnet and Andrew Blake,Poisson image editing, ACM Transactions on Graphics,Vol. 22(3), pp. 313–318

[8] Alexei Efros and Thomas Leung, Texture synthesis by non-parametric sampling, In Proceeding of ICCV, pp. 1033–1038

[9] Felix Manke and Burkhard Wunsche, Analysis of appear-ance space attributes for texture synthesis and morphing,Image and Vision Computing New Zealand 2009. IVCNZ09, pp. 85–90

[10] Sylvain Lefebvre and Hugues Hoppe, Appearance-spacetexture synthesis, ACM SIGGRAPH 2006 Papers, pp. 541–548

[11] Canny J, A Computational Approach To Edge Detection,IEEE Trans. Pattern Analysis and Machine Intelligence1986, Vol. 8(6), pp. 679698

Fig. 5. a) The input image. Image inpaintingresults obtained using the algorithms from: (b)(Bertalmio et al. 2000), (c) (Telea et al. 2004),(d) (Criminisi et al. 2003) and (e) our method.