minicourse - pde techniques for image inpainting part igkadmin/daten/minicourse_part1.pdf ·...

105
Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Sch ¨ onlieb Institute for Numerical and Applied Mathematics University of G¨ ottingen ottingen - January, 7th 2010 Sch ¨ onlieb (NAM, G ¨ ottingen) PDEs for Image Inpainting Part I ottingen-7. January 2010 1 / 50

Upload: others

Post on 28-Oct-2019

17 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Minicourse - PDE Techniques for Image InpaintingPart I

Carola-Bibiane Schonlieb

Institute for Numerical and Applied MathematicsUniversity of Gottingen

Gottingen - January 7th 2010

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 1 50

Outline

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50

Outline

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50

Digital Image Processing

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50

Digital Image Processing Examples

Imaging tasks1

Image Denoising

dArr

CartoonTexture Decomposition

dArr

1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50

Digital Image Processing Examples

Imaging tasks (cont)

Image Segmentationa

aExample from C Guyader LVese (2008)

Image Inpaintinga

dArr

aExample from Bertalmio et al(1998)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 2: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Outline

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50

Outline

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50

Digital Image Processing

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50

Digital Image Processing Examples

Imaging tasks1

Image Denoising

dArr

CartoonTexture Decomposition

dArr

1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50

Digital Image Processing Examples

Imaging tasks (cont)

Image Segmentationa

aExample from C Guyader LVese (2008)

Image Inpaintinga

dArr

aExample from Bertalmio et al(1998)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 3: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Outline

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50

Digital Image Processing

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50

Digital Image Processing Examples

Imaging tasks1

Image Denoising

dArr

CartoonTexture Decomposition

dArr

1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50

Digital Image Processing Examples

Imaging tasks (cont)

Image Segmentationa

aExample from C Guyader LVese (2008)

Image Inpaintinga

dArr

aExample from Bertalmio et al(1998)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 4: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50

Digital Image Processing Examples

Imaging tasks1

Image Denoising

dArr

CartoonTexture Decomposition

dArr

1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50

Digital Image Processing Examples

Imaging tasks (cont)

Image Segmentationa

aExample from C Guyader LVese (2008)

Image Inpaintinga

dArr

aExample from Bertalmio et al(1998)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 5: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Imaging tasks1

Image Denoising

dArr

CartoonTexture Decomposition

dArr

1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50

Digital Image Processing Examples

Imaging tasks (cont)

Image Segmentationa

aExample from C Guyader LVese (2008)

Image Inpaintinga

dArr

aExample from Bertalmio et al(1998)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 6: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Imaging tasks (cont)

Image Segmentationa

aExample from C Guyader LVese (2008)

Image Inpaintinga

dArr

aExample from Bertalmio et al(1998)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 7: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Image inpainting - restore images of different kind

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 8: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Image inpainting - restore images of different kind(cont)

1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 9: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Image inpainting - in general the task is

a

asource Chan and Shen2005

Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 10: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 11: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 12: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications

Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between

Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)

Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists

Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)

Physcial Restoration of the FrescoesrArr Comparison with the digital result

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 13: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 14: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 15: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 16: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Preliminary ResultsGiven image Restored Image

Unfortunately this doesnt always work as straightforward as in this example

Large gaps

Lack of grayvalue contrast

Low color saturation and hue

we need more sophisticated algorithms to restore the frescoes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 17: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Results with Cahn-Hilliard inpainting

2

2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 18: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 19: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Road Reconstruction3

Our data consists of satellite images of roads in Los Angeles of the

following kind

3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 20: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain road

Results

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 21: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

Some applications (cont)

Road Recontruction with Bitwise Cahn-Hilliard

Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 22: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 23: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples)

works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 24: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains

performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 25: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way

and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 26: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Examples

We seek for a method which

smoothly continues image contents into the missing domain

connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 27: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 28: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing What is a Digital Image and how do we Process it

Digital images

A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 29: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 30: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing What is a Digital Image and how do we Process it

Digital images (cont)

it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement

grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 31: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV images

Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 32: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearlets

Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 33: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields

Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 34: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255

Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 35: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 36: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Image models and representations

Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)

Deterministic image models eg generalized functions Lp

functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 37: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions

Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as

D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω

An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)

u φ rarr (u φ)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 38: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)Properties

sensing is linear

with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st

(u φ) =int

Ω

φ(x) dmicro

notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that

(v φ) = (minus1)α+β(u partαpartβφ

) forallφ isin D(Ω)

sensing of distributions mimics the digital sensor devices in CCDcameras

very general class of functions

to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 39: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 40: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Distributions (cont)

Example 1 - Bright spot as delta distribution

Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen

(u φ) = φ(0) for any sensor φ isin D(Ω)

Example 2 - Step edge as Heaviside function

Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie

H(t) =

255 t ge t1

0 t lt t1

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 41: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as

Lp(Ω) =

u int

Ω

|u(x)|p dx lt infin

For p = infin an Linfin image is understood as an essentially bounded functionProperties

Banach spaces with norms

up =[int

Ω

|u(x)|p dx

]1p

Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)

but they carry more structures than gen-eral distributions cf layer-cake representa-tion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 42: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1

H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 43: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 44: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image information

Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 45: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - Hk functions

An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product

(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2

and corresponding norm

uH1 =radicu2L2 + nablau2L2timesL2

Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 46: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions

For a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 47: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (TV)

For u isin L1loc(Ω)

V (u Ω) = supint

Ωunabla middot ϕ dx ϕ isin

[C1

c (Ω)]2

ϕinfin le 1

is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)

hArrV (u Ω) lt infin

In such a case|Du|(Ω) = V (u Ω)

where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to

|Du|(Ω) =intΩ|nablau| dx

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 48: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 49: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Deterministic image models - The total variation (cont)

What does it measure

Figure The total variation measures the size of the jump

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 50: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 51: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 52: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Mathematical Image Models

Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be

Γλ = x isin Ω u(x) = λ

Classical level line representation is the one-parameter family of all the levellines ie

Γλ λ isin R

Coarea formula (weak version) For a smooth image u we haveintΩ

|nablau| dx =int infin

minusinfinlength(Γλ) dλ

a similar relation holds for functions of bounded variation (length has to beredefined)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 53: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 54: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering

Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution

u = (Kσ lowast f)(x) =int

R2

Kσ(xminus y)f(y) dy

where Kσ denotes the 2D Gaussian of width σ gt 0 ie

Kσ(x) =1

2πσ2eminus|x

2|(2σ2)

PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 55: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 56: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Gaussian filtering (cont)

Behaviour in the frequency domain

(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)

= eminus|w|2(2σ2) middot (Ff)(w)

Low-pass filter that attenuates high frequencies

Equivalence to linear diffusion filtering

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 57: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 58: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing

For a given f isin C(R2) we consider the linear diffusion prozess

ut = ∆u

u(x 0) = f(x)

with solution

u(x) =

f(x) t = 0(Kradic

2t lowast f)(x) t gt 0

PropertiesSolution is unique in the class of functions u with

|u(x t)| le M middot ea|x|2 (Ma gt 0)

The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle

infR2

f le u(x t) le supR2

f on R2 times [0infin)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 59: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image smoothing (cont)

Gaussian smoothing structures of order σ requires to stop thediffusion process at time

T =12σ2

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50

reka500heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 60: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges

ut = div(g(|nablau|2)nablau

)

where g is called the diffusivity of the process eg

g(s2) =1

1 + s2δ2(δ gt 0)

Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50

reka300nonlineardiffavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 61: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting

Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50

adameva250heatequavi
Media File (videoavi)

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 62: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λ

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 63: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion

The heat equation for image inpainting (cont)

Let D sub Ω be a hole in the image domain We solve

ut = ∆u in D

u(x 0) = 0 in D

u|partD = f |partD

or we solve for the union of all holes D

ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω

where

χΩD(x) =

1 x isin Ω D

0 otherwise

and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 64: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting

Outline

1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion

2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 65: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Some history and classifications

The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 66: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 67: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches

Let us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 68: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Some history and classifications (cont)

PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]

In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 69: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Bertalmio et alrsquos approach 2000

Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings

Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE

ut = nablaperpu middot nabla∆u

solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 70: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 71: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation

Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy

E(γ) =int

γ

(a + bκ2) ds

where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)

Basic principle prolongate level lines by minimizing their length andcurvature

transformed into a functional approach by Chan amp Shen Eulers elasticainpainting

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 72: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting State of the Art Methods

Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model

Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise

ut = λnabla middot(nablau

|nablau|

)+ χΩD(f minus u)

Disadvantage level lines are interpolated linearly

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 73: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω D

R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 74: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport

The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 75: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 76: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

The variational approach

J (u) = R(u) +12λ

∥∥χΩD(uminus g)∥∥2

B1rarr min

uisinB2

where

χΩD(x) =

1 Ω D

0 D

is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD

Now for B1 = L2(Ω) we also have

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 77: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 78: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 79: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

PDE approach

the corresponding Euler-Lagrange equation

minusλnablaR(u) + χΩD(g minus u) = 0 in Ω

the corresponding steepest descent equation for u( t = 0) = g isthe given image

ut = minusλnablaR(u) + χΩD(g minus u) in Ω

in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 80: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches

no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 81: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary

no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 82: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 83: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

VariationalPDE approach - some remarks

In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region

These properties make such inpainting approaches applicable to allkinds of damaged images

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 84: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 85: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)

R(u) =intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 86: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002)

Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 87: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life

rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 88: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon

more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 89: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models

seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 90: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting

Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 91: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)

TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 92: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 93: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting The VariationalPDE Approach

Inpainting review

R(u) =intΩ |nablau|2 dx Harmonic-inpainting

R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =

intΩ(1 + (nabla middot (nablau|nablau|))2)|nablau| dx Euleracutes elastica

inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)

so why do we consider higher-order inpainting models

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 94: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 95: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 96: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λ

Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 97: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -

Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 98: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Second order approaches

Example TV-inpainting

|Du|(Ω) asympint

Ω|nablau| dx

Propagate sharp edges into the damaged domain +

minu

intΩ|nablau| dx lArrrArr min

Γλ

int infin

minusinfinlength(Γλ) dλ

where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 99: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(a) Connectivity principle (b) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 100: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

Example Eulerrsquos elastica inpainting

minu

intΩ(a + b

(nabla middot

(nablau|nablau|

))2)|nablau| dx

lArrrArrminΓλ

intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ

(c) Connectivity principle (d) Smooth continuation

4

4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 101: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2

[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 102: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches (cont)

Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of

ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω

where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves

∆(ε∆uminus 1εF prime(u)) = 0 in D

u = f on partD

nablau = nablaf on partD

for f regular enough (f isin C2)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 103: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Higher order approaches

ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 104: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

Next time

Cahn-Hilliard inpainting

ut = ∆(minusε∆u +1

εF prime(u)) +

1

λχΩD(f minus u) in Ω

(existence stationary solutions characterization of solutions boundaryconditions)

TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)

ut = ∆p +1

λχΩD(f minus u) p isin partTV (u)

(Stationary solution cooperation of transport and diffusion error analysis)

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting
Page 105: Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute

Image Inpainting Second- Versus Higher-Order PDEs for Inpainting

End of Part I

Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk

Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50

  • Outline
  • Main Talk
    • Digital Image Processing
      • Examples
      • What is a Digital Image and how do we Process it
      • Mathematical Image Models
      • Gaussian Filtering the Heat Equation and Nonlinear Diffusion
        • Image Inpainting
          • State of the Art Methods
          • The VariationalPDE Approach
          • Second- Versus Higher-Order PDEs for Inpainting