examplar-based inpainting

Examplar-basedinpainting

O. Le Meur

Inpainting:context andissues

Examplar-basedinpaintingPresentationNotationCriminisi et al.smethod

Variants ofCriminisismethodFilling ordercomputationTexture synthesisSome examplesLimitations

Super-resolution-based inpaintingmethodProposed approachMore than oneinpaintingLoopy BeliefPropagationSuper-resolution

Results andcomparison withexisting methodsResultsComparison withexisting methods

Conclusion

Examplar-based inpainting

Olivier Le [email protected]

IRISA - University of Rennes 1

June 19, 20141 / 44


O. Le Meur






Conclusion

Inpainting: context and issues (1/3)

This talk is about inpainting. We will heavily rely upon these papers:

C. Guillemot & O. Le Meur, Image inpainting: overviewand recent advances, IEEE Signal Processing Magazine,Vol. 1, pp. 127-144, 2014.

O. Le Meur, M. Ebdelli and C. Guillemot, Hierarchicalsuper-resolution-based inpainting, IEEE TIP, vol.22(10), pp. 3779-3790, 2013.

O. Le Meur & C. Guillemot, Super-resolution-basedinpainting, ECCV 2012.

2 / 44


O. Le Meur






Conclusion


InpaintingInpainting corresponds to filling holes (i.e. missing areas) in im-ages (Bertalmio et al., 2000).

Let be an image I defined as

I : Rn 7 Rm

Let be a degradation operator M

M : 7 {0, 1}

M (x) ={

0, if x U1, otherwise

Let F the observed image:

F = M I

n = 2 for a 2D imagem = 3 for (R,G,B) image

= S U, S being the known part

of I U the unknown part of I

is the Hadamard product

3 / 44


O. Le Meur






Conclusion


Different configurations according to the definition of M :

Original image80% of the pixels

have beenremoved.

damaged portionsin black, scratches object removal

Sparsity andlow-rank methods

Diffusion-basedmethods

Examplar-basedmethods

4 / 44


O. Le Meur






Conclusion

Outline of the presentation

1 Inpainting: context and issues

2 Examplar-based inpainting

3 Variants of Criminisis method

4 Super-resolution-based inpainting method

5 Results and comparison with existing methods

6 Conclusion

5 / 44


O. Le Meur






Conclusion



2 Examplar-based inpaintingI PresentationI NotationI Criminisi et al.s method




6 Conclusion

6 / 44


O. Le Meur






Conclusion

Examplar-based inpainting (1/4)

Texture synthesisExamplar-based inpainting methods rely on the assumption that theknown part of the image provides a good dictionary which could beused efficiently to restore the unknown part (Efros and Leung, 1999).

The recovered texture is thereforelearned from similar regions. This can be done simply by

sampling, copying orcombining patches from theknown part of the image;

Template Matching Patches are then stitched

together to fill in the missingarea.

7 / 44


O. Le Meur






Conclusion


Notations: a patch px is a discretized

N N neighborhoodcentered on the pixel px .This patch can be vectorizedin a raster-scan order as amN 2-dimensional vector;

ukpx denotes the unknownpixels of the patch;

kpx denotes its knownpixels;

px(i) denotes the ithnearest neighbour of px ;

U is the front line;

8 / 44


O. Le Meur






Conclusion


Criminisi et al.s algorithmCriminisi et al. (Criminisi et al., 2004) has brought a new momentumto inpainting applications and methods. They proposed a new methodbased on two sequential stages:

1 Filling order computation;2 Texture synthesis.

1 Filling order computation: P(px) = C (px)D(px)Confidence term

C (px) =

qkpx C (q)|px |

where |px | is the area of px .

Data term

D(px) =|I(px) npx |

where is a normalizationconstant in order to ensure thatD(px) is in the range 0 to 1.

9 / 44


O. Le Meur






Conclusion


2 Texture synthesis:

A template matching is performed within a local neighborhood:

py = arg minqW d(kpq ,

kpx )

W S is the window search; kpx are the known pixels of the patch px with the highest

priority; kpy are the known pixels of the nearest patch neighbor; d(a, b) is the sum of squared differences between patches a and

b.

The pixels of the patch ukpy are then copied into the unknown pixelsof the patch px .

10 / 44


O. Le Meur






Conclusion




3 Variants of Criminisis methodI Filling order computationI Texture synthesisI Some examplesI Limitations



6 Conclusion11 / 44


O. Le Meur






Conclusion

Filling order computation (1/4)

P(px) = C (px)D(px)

Two variants are here presented: Tensor-based data term (Le Meur et al., 2011);

Sparsity-based data term (Xu and Sun, 2010).

Many others: edge-based data term, transformation of the data termin a nonlinear fashion, entropy-based data term...

12 / 44


O. Le Meur






Conclusion


Tensor-based data termInstead of using the gradient, (Le Meur et al., 2011) used the structuretensor which is more robust:

D(px) = + (1 )exp( (1 2)2

)where is a positive value and [0, 1].The structure tensor is a symmetric, positive semi-definitematrix (Weickert, 1999):

J, [I ] = K ( mi=1(Ii K)(Ii K)T

)

where Ka is a Gaussian kernel with a standard deviation a. Theparameters and are called integration scale and noise scale,respectively.

13 / 44


O. Le Meur






Conclusion


D(px) = + (1 )exp( (1 2)2

)

When 1 ' 2, the data term tends to . It tends to 1 when1 >> 2.

14 / 44


O. Le Meur






Conclusion


Sparsity-based data termSparsity-based data term (Xu and Sun, 2010) is based on the sparse-ness of nonzero patch similarities:

D(px) =

|Ns(px)||N (px)| pjWs w2px ,pjwhere Ns and N are the numbers of valid and candidate patches inthe search window.Weight wpx ,pj is proportional to the similarity between the two patchescentered on px and pj (

j wpx ,pj = 1).

A large value of the structure sparsity term means sparse similaritywith neighboring patches a good confidence that the input patch is on some structure.

15 / 44


O. Le Meur






Conclusion

Texture synthesis (1/4)

Texture synthesis with more than one candidateFrom K patches px(i) which are the most similar to the known partkpx of the input patch, the unknown part of the patch to be filled ukpxis then obtained by a linear combination of the sub-patches ukpx(i) .

ukpx =Ki=1

wiukpx(i)

How can we compute the weightswi of this linear combination?

Note: K is locally adjusted by usingan -ball including patches within acertain radius.

16 / 44


O. Le Meur






Conclusion


ukpx =Ki=1

wiukpx(i)

Different solutions exist (Guillemot et al., 2013): Average template matching: wi = 1K , i; Non-local means approach (Buades et al., 2005):

wi = exp(d(pkx , pkx(i))

h2

)

Least-square method minimizing

E(w) = kpx Aw22,aw = arg min

wE(w)

17 / 44


O. Le Meur






Conclusion


ukpx =Ki=1

wiukpx(i)

Constrained Least-square optimization with the sum-to-oneconstraint of the weight vector LLE method (Saul andRoweis, 2003)

E(w) = kpx Aw22,aw = arg min

wE(w) s.t. wT1K = 1

Constrained Least-square optimization with positive weights NMF method (Lee and Seung, 2001)

w = arg minw

E(w) s.t. wi 0

18 / 44


O. Le Meur






Conclusion


Similarity metrics: Using a Gaussian weighted Euclidean distance

dL2(px , py ) = px py22,awhere a controls the decay of the Gaussian functiong(k) = e

|k|2a2 , a > 0;

A better distance introduced in (Bugeau et al., 2010, Le Meurand Guillemot, 2012):

d(px , py ) = dL2(px , py ) (1 + dH (px , py ))where dH (px , py ) is the Hellinger distance

dH (px , py ) =

1k

p1(k)p2(k)

where p1 and p2 represent the histograms of patches px , py ,respectively.

19 / 44


O. Le Meur






Conclusion

Some Examples (1/2)

Inpainted pictures with (Criminisi et al., 2004)s method (Courtesy ofP. Perez):

20 / 44

Some Examples (2/2)

Results from (Le Meur et al., 2011).


O. Le Meur






Conclusion

Limitations

Very sensitive to the parameter settings such as the filling orderand the patch size:

Examplar-based methods are a one-pass greedy algorithms.

22 / 44


O. Le Meur






Conclusion





4 Super-resolution-based inpainting methodI Proposed approachI More than one inpaintingI Loopy Belief PropagationI Super-resolution


6 Conclusion23 / 44


O. Le Meur






Conclusion

Proposed approach (1/1)

Objectives of the proposed methodWe apply an examplar-based inpainting algorithm several times andfuse together the inpainted results.

less sensitive to the inpainting setting;relax the greedy constraint.

The inpainting method is applied on a coarse version of the inputpicture:

less demanding of computational resources;less sensitive to noise;K candidates for the texture synthesis without introducing blur.

Need to fuse the inpainted images and to retrieve the highestfrequencies

Loopy Belief Propagation and Super-Resolution algorithms.24 / 44


O. Le Meur






Conclusion

More than one inpainting (1/1)

The baseline algorithm is anexamplar-based method: Filling order

computation; Texture synthesis.

Decimation factor n = 3 13 sets of parameters

Table: Thirteen inpainting configurations.Setting Parameters

1Patchs size 5 5

Decimation factor n = 3Search window 80 80Sparsity-based filling order

2 default + rotation by 180 degrees3 default + patchs size 7 74 default + rotation by 180 degrees+ patchs size 7 75 default + patchs size 11 116 default + rotation by 180 degrees+ patchs size 11 117 default + patchs size 9 98 default + rotation by 180 degrees+ patchs size 9 99 default + patchs size 9 9+ Tensor-based filling order

10 default + patchs size 7 7+ Tensor-based filling order11 default + patchs size 5 5+ Tensor-based filling order12 default + patchs size 11 11+ Tensor-based filling order

13default + rotation by 180 degrees

+ patchs size 9 9+ Tensor-based filling order

25 / 44


O. Le Meur






Conclusion

Loopy Belief Propagation (1/5)

. . . . . .Loopy Belief Propagation is used to fuse together the 13 inpainted

images.

Let be a finite set of labels L composed of M = 13 values.

E(l) =p

Vd(lp) +

(n,m)N4Vs(ln, lm)

where, lp the label of pixel px , represents the pixel in U and N4 isa neighbourhood system. is a weighting factor.

26 / 44


O. Le Meur






Conclusion


E(l) =p

Vd(lp) +

(n,m)N4Vs(ln, lm)

Vd(lp) represents the cost of assigning a label lp to a pixel px :

Vd(lp) =nL

u

{I (l)(x + u) I (n)(x + u)

}2 Vs(ln, lm) is the discontinuity cost:

Vs(ln, lm) = (ln lm)2

The minimization is performed iteratively (less than 15iterations) (Boykov and Kolmogorov, 2004, Boykov et al., 2001,Yedidia et al., 2005).

27 / 44


O. Le Meur






Conclusion


LBP convergence: 13 inpainted image in

input; 25 iterations; resolution=80 120.

28 / 44


O. Le Meur






Conclusion




29 / 44


O. Le Meur






Conclusion




30 / 44


O. Le Meur






Conclusion

Super-resolution (1/2)

For the LR patch correspondingto the HR patch having thehighest priority: We look for its best

neighbour; Only the best candidate is

kept; The corresponding HR

patch is simply deduced. Its pixel values are then

copied into the unknownparts of the current HRpatch.

31 / 44


O. Le Meur






Conclusion

Super-resolution (2/2)

To speed-up the process, we can perform thesearch:

within a search window;

within a dictionary (as illustrated on theright) composed of LR patches withtheir corresponding HR patches.

32 / 44


O. Le Meur






Conclusion






5 Results and comparison with existing methodsI ResultsI Comparison with existing methods

6 Conclusion

33 / 44


O. Le Meur






Conclusion

Results (1/4)

34 / 44


O. Le Meur






Conclusion

Results (2/4)

35 / 44


O. Le Meur






Conclusion

Results (3/4)

36 / 44


O. Le Meur






Conclusion

Results (4/4)

37 / 44


O. Le Meur






Conclusion

Comparison with existing methods (1/5)

Three methods have been tested:

[Komodakis] N. Komodakis, and G. Tziritas, Image Completionusing Global Optimization. in CVPR 2007 (Komodakis andTziritas, 2007);

[Pritch] Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-Map ImageEditing. in ICCV 2009 (Pritch et al., 2009);

[He] K. He and J. Sun, Statistics of Patch Offsets for ImageCompletion. in ECCV 2012 (He and Sun, 2012).

38 / 44


O. Le Meur






Conclusion


From left to right: Komodakis, Pritch, He, Ours.39 / 44


O. Le Meur






Conclusion


From left to right: Komodakis, Pritch, He, Ours.40 / 44


O. Le Meur






Conclusion


Much more results on the link:http://people.irisa.fr/Olivier.Le_Meur/publi/2013_TIP/

indexSoA.html

41 / 44


O. Le Meur






Conclusion


Limitations and failure cases:

From left to right: original, Hes method and proposed one.

No semantic information are used... No objective quality metric.

42 / 44


O. Le Meur






Conclusion







6 Conclusion

43 / 44


O. Le Meur






Conclusion

Conclusion

A new framework to perform inpainting of still color pictures:coarse inpainting + super-resolution.Binary file could be downloaded:http://people.irisa.fr/Olivier.Le_Meur/publi/2013_TIP/index.html

A natural extension is to deal with video inpainting.A paper dealing with video inpainting under revision in IEEE TIP.

44 / 44


O. Le Meur

References

ReferencesM. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. In SIGGRPAH 2000, 2000.Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision.

IEEE Trans. On PAMI, 26(9):11241137, 2004.Y. Boykov, O. Veksler, and R.Zabih. Efficient approximate energy minimization via graph cuts. IEEE Trans. On PAMI, 20(12):

12221239, 2001.A. Buades, B. Coll, and J.M. Morel. A non local algorithm for image denoising. In IEEE Computer Vision and Pattern Recognition

(CVPR), volume 2, pages 6065, 2005.A. Bugeau, M. Bertalmo, V. Caselles, and G. Sapiro. A comprehensive framework for image inpainting. IEEE Trans. on Image

Processing, 19(10):26342644, 2010.A. Criminisi, P. Perez, and K. Toyama. Region filling and object removal by examplar-based image inpainting. IEEE Trans. On

Image Processing, 13:12001212, 2004.A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In IEEE Computer Vision and Pattern Recognition

(CVPR), pages 10331038, 1999.C. Guillemot, M. Turkan, O. Le Meur, and M. Ebdelli. Object removal and loss concealment using neigbor embedding methods.

Signal processing: image communication, 28:14051419, 2013.K. He and J. Sun. Statistics of patch offsets for image completion. In ECCV, 2012.N. Komodakis and G. Tziritas. Image completion using efficient belief propagation via priority scheduling and dynamic pruning.

IEEE Trans. On Image Processing, 16(11):2649 2661, 2007.O. Le Meur and C. Guillemot. Super-resolution-based inpainting. In ECCV, pages 554567, 2012.O. Le Meur, J. Gautier, and C. Guillemot. Examplar-based inpainting based on local geometry. In ICIP, 2011.D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In In NIPS, pages 556562. MIT Press, 2001.Y. Pritch, E. Kav-Venaki, and S. Peleg. Shift-map image editing. In ICCV09, pages 151158, Kyoto, Sept 2009.L.K. Saul and S.T. Roweis. Think globally, fit locally: Unsupervised learning of low dimensional manifolds. Journal of Machine

Learning Research, 4:119155, 2003.J. Weickert. Coherence-enhancing diffusion filtering. International Journal of Computer Vision, 32:111127, 1999.Z. Xu and J. Sun. Image inpainting by patch propagation using patch sparsity. IEEE Trans. on Image Processing, 19(5):

11531165, 2010.J.S. Yedidia, W.T. Freeman, and Y. Weiss. Constructing free energy approximations and generalized belief propagation algorithms.

IEEE Transactions on Information Theory, 51:22822312, 2005.44 / 44

Inpainting: context and issuesExamplar-based inpaintingPresentationNotationCriminisi et al.'s method

Variants of Criminisi's methodFilling order computationTexture synthesisSome examplesLimitations

Super-resolution-based inpainting methodProposed approachMore than one inpaintingLoopy Belief PropagationSuper-resolution

Results and comparison with existing methodsResultsComparison with existing methods

Conclusion*

0.0: 0.1: 0.2: 0.3: 0.4: 0.5: 0.6: 0.7: 0.8: 0.9: 0.10: 0.11: 0.12: 0.13: 0.14: 0.15: 0.16: 0.17: 0.18: 0.19: 0.20: 0.21: 0.22: 0.23: 0.24: 0.25: 0.26: 0.27: 0.28: 0.29: 0.30: 0.31: 0.32: 0.33: 0.34: 0.35: 0.36: 0.37: 0.38: 0.39: 0.40: 0.41: 0.42: 0.43: 0.44: 0.45: 0.46: 0.47: 0.48: 0.49: 0.50: 0.51: 0.52: 0.53: 0.54: 0.55: 0.56: 0.57: 0.58: 0.59: 0.60: 0.61: 0.62: 0.63: 0.64: 0.65: 0.66: 0.67: 0.68: 0.69: 0.70: 0.71: 0.72: 0.73: 0.74: 0.75: 0.76: 0.77: 0.78: 0.79: 0.80: 0.81: 0.82: 0.83: 0.84: 0.85: 0.86: 0.87: 0.88: 0.89: 0.90: 0.91: 0.92: 0.93: 0.94: 0.95: 0.96: 0.97: 0.98: 0.99: 0.100: 0.101: 0.102: 0.103: 0.104: 0.105: 0.106: 0.107: 0.108: 0.109: 0.110: 0.111: 0.112: 0.113: 0.114: 0.115: 0.116: 0.117: 0.118: 0.119: 0.120: 0.121: 0.122: 0.123: 0.124: anm0: 1.0: 1.1: 1.2: 1.3: 1.4: 1.5: 1.6: 1.7: 1.8: 1.9: 1.10: 1.11: 1.12: 1.13: 1.14: 1.15: 1.16: 1.17: 1.18: 1.19: 1.20: 1.21: 1.22: 1.23: 1.24: anm1: 2.0: 2.1: 2.2: 2.3: 2.4: 2.5: 2.6: 2.7: 2.8: 2.9: 2.10: 2.11: 2.12: 2.13: 2.14: 2.15: 2.16: 2.17: 2.18: 2.19: 2.20: 2.21: 2.22: 2.23: 2.24: anm2: 3.0: 3.1: 3.2: 3.3: 3.4: 3.5: 3.6: 3.7: 3.8: 3.9: 3.10: 3.11: 3.12: 3.13: 3.14: 3.15: 3.16: 3.17: 3.18: 3.19: 3.20: 3.21: 3.22: 3.23: 3.24: anm3: 4.0: 4.1: 4.2: 4.3: 4.4: 4.5: 4.6: 4.7: 4.8: 4.9: 4.10: 4.11: 4.12: 4.13: 4.14: 4.15: 4.16: 4.17: 4.18: 4.19: 4.20: 4.21: 4.22: 4.23: 4.24: anm4: 5.0: 5.1: 5.2: 5.3: 5.4: 5.5: 5.6: 5.7: 5.8: 5.9: 5.10: 5.11: 5.12: 5.13: 5.14: 5.15: 5.16: 5.17: 5.18: 5.19: 5.20: 5.21: 5.22: 5.23: 5.24: anm5: 6.0: 6.1: 6.2: 6.3: 6.4: 6.5: 6.6: 6.7: 6.8: 6.9: 6.10: 6.11: 6.12: 6.13: 6.14: 6.15: 6.16: 6.17: 6.18: 6.19: 6.20: 6.21: 6.22: 6.23: 6.24: anm6: 7.0: 7.1: 7.2: 7.3: 7.4: 7.5: 7.6: 7.7: 7.8: 7.9: 7.10: 7.11: 7.12: 7.13: 7.14: 7.15: 7.16: 7.17: 7.18: 7.19: 7.20: 7.21: 7.22: 7.23: 7.24: 7.25: 7.26: 7.27: 7.28: 7.29: 7.30: 7.31: 7.32: 7.33: 7.34: 7.35: 7.36: 7.37: 7.38: 7.39: 7.40: 7.41: 7.42: 7.43: 7.44: 7.45: 7.46: 7.47: 7.48: 7.49: 7.50: 7.51: 7.52: 7.53: 7.54: 7.55: 7.56: 7.57: 7.58: 7.59: 7.60: 7.61: 7.62: 7.63: 7.64: 7.65: 7.66: 7.67: 7.68: 7.69: 7.70: 7.71: 7.72: 7.73: 7.74: 7.75: 7.76: 7.77: 7.78: 7.79: 7.80: 7.81: 7.82: 7.83: 7.84: 7.85: 7.86: 7.87: 7.88: 7.89: 7.90: 7.91: 7.92: 7.93: 7.94: 7.95: 7.96: 7.97: 7.98: 7.99: 7.100: 7.101: 7.102: 7.103: 7.104: 7.105: 7.106: 7.107: 7.108: 7.109: 7.110: 7.111: 7.112: 7.113: 7.114: 7.115: 7.116: 7.117: 7.118: 7.119: 7.120: 7.121: 7.122: 7.123: 7.124: 7.125: 7.126: 7.127: 7.128: 7.129: 7.130: 7.131: 7.132: 7.133: 7.134: 7.135: 7.136: 7.137: 7.138: 7.139: 7.140: 7.141: 7.142: 7.143: 7.144: 7.145: 7.146: 7.147: 7.148: 7.149: 7.150: 7.151: 7.152: 7.153: anm7:

examplar-based inpainting

Engineering