Application into Image Inpainting

Criminisi, A.; Perez, P.; Toyama, K., "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. on Image Proc., vol.13, no.9, pp. 1200- 1212, Sept. 2004

At the Intersection

Error Concealment, Image Recovery … …

Importance of Scanning Order

Heuristic Rules

Green/red color: high (low) priority pixels

Geometric cue: ConvexityPhotometric cue: gradient

Image Inpainting

SpecificScanning

order

Repair byNonparametric

samplingImage torepaired

Repairedimage

Synthetic Image Example

Kanisza Triangle

FinalIntermediateInitial

Real-world Image Example

Computer Assignment 5 Implement Efros-Leung’s scheme Test it on image inpainting (you

might want to try different textures) When does it work and when does it

fail? Computational complexity reduction

Open Questions Where to inpaint? Sequential vs. parallel inpainting Performance evaluation

Seeing is not always believing Failure examples

Few researchers report them Comparison with other approaches

Why does A work better than B?

Error Concealment in Image Communication

videoimage

Sequential Parallel

Why Parallel? Avoid choosing scanning order and its

associated error propagation problem Modeling multimedia signal at a patch

(e.g., speech frame, image block) level instead of data (e.g., speech sample, image pixel) level

Parallelism has to work with distributed processing together (redundancy is exploited to fight against uncertainty)

Why Nonlocal?

B4

B2

B3

B0

B1

RPP Local neighborsin range (photometricsimilarity)

nonlocal patchesin domain (geometricproximity)

Algorithm Flowchart

Projectiononto

observationconstraint

Projectionontoprior

constraint

Data clustering

in Rp

Coringalgorithm

Better estimate of local neighborhood in the patch space

Inner loopOuter loop

Better estimate of missing data in the time domain

Sparsity Enforcement

B0 BkB1 Pack into3D Array D 3D-FFT

Thresholding

…

3D-IFFTPack into

3D Array DB0 BkB1 …^ ^ ^

Projectionontoprior

constraint

Image Examples

27.06dB 29.14dB 30.16dB

original damagedexemplar

basedGuleryuz’s

schemeour

work

18.06dB 17.57dB

35.72dB 17.93dB 37.37dB

22.36dB

26.75dB 24.63dB 31.84dB

original damagedexemplar

basedGuleryuz’s

schemeour

work

26.52dB 25.84dB 33.08dB

28.78dB 28.09dB 34.26dB

21.21dB 14.66dB

17.83dB 17.66dB

26.58dB

20.24dB

Parallel and Distributed Processing (PDP) Dominant form of

Connectionism (models mental or behavioral phenomena as the emergent processes of interconnected networks of simple units.)

More widely known as neural networks since 1980s

Image Quilting for Texture Synthesis &

TransferAlexei Efros and

Bill Freeman (SIGGRAPH’2001)

++ ==

Pixel-level Nonparametric Sampling

Assuming Markov property, compute P(p|N(p)) Building explicit probability tables infeasible Instead, let’s search the input image for all similar

neighborhoods — that’s our histogram for p To synthesize p, just pick one match at random

pp

non-parametricsampling

Input image Synthesizing a pixel

Block-level Chaos Mosaic

Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges

input

idea result

Comparison with Periodic Mosaic

periodic chaotic

input

result

Doesn’t work for structured textures

pp

Image Quilting=Nonparametric Sampling+Chaos Mosaic

Observation: neighbour pixels are highly correlated

Input image

non-parametricsampling

BB

Idea:Idea: unit of synthesis = block unit of synthesis = block• Exactly the same but now we want P(B|N(B))• Much faster: synthesize all pixels in a block at once• Not the same as multi-scale!

Synthesizing a block

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

min. error boundary

Minimal error boundaryoverlapping blocks vertical boundary

__ ==22

overlap error

Search for the Shortest Path

start

end

An Elegant Solution Exhaustive search is

computationally infeasible Fortunately, there exists a fast and

optimum solution known as Dynamic Programming (DP) in Operations Research and Viterbi Algorithm (VA) in communication

DP/VA Example

start end

Basic Idea: Only keep the best Q survivors in sequential decision

DP/VA Step-by-Step Result

Image Quilting Algorithm Pick size of block and size of overlap Synthesize blocks in raster order

Search input texture for block that satisfies minimum error constraints (above and left)

Paste new block into resulting texture

++ ==

++ ==

++ ==

parmesan

rice

Summary for Texture Synthesis Parametric approach

Histogram/high order statistics matching in the wavelet/steerable pyramid space

Nonparametric approach Conceptually simple and effective

Interesting application into image inpainting & quilting

Produce visually convincing results Parametric vs. nonparametric: which is the

winner? It depends on the application How to combine them together?